zrhoffman opened a new issue #4812: URL: https://github.com/apache/trafficcontrol/issues/4812
<!-- ************ STOP!! ************ If this issue identifies a security vulnerability, DO NOT submit it! Instead, contact the Apache Software Foundation Security Team at [email protected] and follow the guidelines at https://www.apache.org/security/ regarding vulnerability disclosure. --> <!-- - For *SUPPORT QUESTIONS*, use the [Traffic Control slack channels](https://traffic-control-cdn.slack.com) or [Traffic Control mailing lists](http://trafficcontrol.apache.org/mailing_lists/). - Before submitting, please **SEARCH GITHUB** for a similar issue or PR. --> ## I'm submitting a ... <!-- (check all that apply with "[x]") --> <!--- security vulnerability (STOP!! - see above)--> - [x] bug report - [ ] new feature / enhancement request - [ ] improvement request (usability, performance, tech debt, etc.) - [ ] other <!--(Please do not submit support requests here - see above)--> ## Traffic Control components affected ... <!-- (check all that apply with "[x]") --> - [ ] CDN in a Box - [ ] Documentation - [ ] Grove - [ ] Traffic Control Client - [x] Traffic Monitor - [ ] Traffic Ops - [ ] Traffic Ops ORT - [ ] Traffic Portal - [ ] Traffic Router - [ ] Traffic Stats - [ ] Traffic Vault - [ ] unknown ## Current behavior: <!-- Describe how the bug manifests / how the current features are insufficient. --> About 50% of the time, Traffic Monitor panics and exits ~20 seconds after starting up. ## Expected / new behavior: <!-- Describe what the behavior would be without the bug / how the feature would improve Traffic Control --> No Traffic Monitor panic ## Minimal reproduction of the problem with instructions: <!-- If the current behavior is a bug or you can illustrate your feature request better with an example, please provide the *STEPS TO REPRODUCE* and include the applicable TC version. --> ## Anything else: <!-- e.g. stacktraces, related issues, suggestions how to fix --> Reproducible about 50% of the time from f2a6b5ba36/#4690 onwards. CDN-in-a-Box Traffic Monitor output (everything before Traffic Monitor starts omitted): ```go trafficmonitor_1 | 2020-06-22T04:14:17.405326652Z + traffic_monitor_command=(/opt/traffic_monitor/bin/traffic_monitor -opsCfg /opt/traffic_monitor/conf/traffic_ops.cfg -config /opt/traffic_monitor/conf/traffic_monitor.cfg) trafficmonitor_1 | 2020-06-22T04:14:17.405411350Z + [[ false == true ]] trafficmonitor_1 | 2020-06-22T04:14:17.405418669Z + /opt/traffic_monitor/bin/traffic_monitor -opsCfg /opt/traffic_monitor/conf/traffic_ops.cfg -config /opt/traffic_monitor/conf/traffic_monitor.cfg trafficmonitor_1 | 2020-06-22T04:14:17.411355728Z INFO: traffic_monitor.go:69: 2020-06-22T04:14:17.411262786Z: Starting with config {CacheHealthPollingInterval:6s CacheStatPollingInterval:6s MonitorConfigPollingInterval:15s HTTPTimeout:2s PeerPollingInterval:5s PeerOptimistic:true PeerOptimisticQuorumMin:0 MaxEvents:200 MaxStatHistory:5 MaxHealthHistory:5 HealthFlushInterval:20ms StatFlushInterval:20ms StatBufferInterval:0s LogLocationError:stdout LogLocationWarning:stdout LogLocationInfo:stdout LogLocationDebug:stdout LogLocationEvent:stdout ServeReadTimeout:10s ServeWriteTimeout:10s HealthToStatRatio:4 StaticFileDir:/opt/traffic_monitor/static/ CRConfigHistoryCount:20000 TrafficOpsMinRetryInterval:100ms TrafficOpsMaxRetryInterval:1m0s CRConfigBackupFile:/opt/traffic_monitor/crconfig.backup TMConfigBackupFile:/opt/traffic_monitor/tmconfig.backup TrafficOpsDiskRetryMax:2 CachePollingProtocol:both PeerPollingProtocol:both} trafficmonitor_1 | 2020-06-22T04:14:17.412896430Z INFO: srvhttp.go:95: 2020-06-22T04:14:17.412795639Z: Starting Web Server trafficmonitor_1 | 2020-06-22T04:14:17.413260439Z INFO: srvhttp.go:142: 2020-06-22T04:14:17.413196972Z: Web server listening on :80 trafficmonitor_1 | 2020-06-22T04:14:17.561460460Z INFO: monitorconfig.go:89: 2020-06-22T04:14:17.561345653Z: MonitorConfigPoller: received new opsConfig: {tmon jhdslvhdfsuklvfhsuvlhs https://trafficops.infra.ciab.test:443 true CDN-in-a-Box :80 } trafficmonitor_1 | 2020-06-22T04:14:17.561484683Z INFO: monitorconfig.go:92: 2020-06-22T04:14:17.561380258Z: MonitorConfigPoller: received new session: {0xc00000e5a8 0xc000608008 {0xc00000e5b0 0xc0001c77c0} {0xc00024dae0 0xc00060a000 0xc00025ae98 0xc0000a6018 0xc000608000} /opt/traffic_monitor/crconfig.backup /opt/traffic_monitor/tmconfig.backup} trafficmonitor_1 | 2020-06-22T04:14:23.347622839Z ERROR: datareq.go:151: 2020-06-22T04:14:23.347474783Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-22T04:14:28.345457713Z ERROR: datareq.go:151: 2020-06-22T04:14:28.345208814Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-22T04:14:32.442416435Z INFO: monitorconfig.go:263: 2020-06-22T04:14:32.442161056Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-22T04:14:32.442483352Z INFO: monitorconfig.go:269: 2020-06-22T04:14:32.442231924Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-22T04:14:32.442497645Z INFO: monitorconfig.go:263: 2020-06-22T04:14:32.442356595Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-22T04:14:32.442511439Z INFO: monitorconfig.go:269: 2020-06-22T04:14:32.442390665Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-22T04:14:32.442660473Z INFO: monitorconfig.go:98: 2020-06-22T04:14:32.442505314Z: MonitorConfigPoller: received new interval: 1.94s trafficmonitor_1 | 2020-06-22T04:14:33.348132134Z ERROR: datareq.go:151: 2020-06-22T04:14:33.347852471Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-22T04:14:36.627379709Z DEBUG: cache.go:183: 2020-06-22T04:14:36.626905961Z: poll 1 2020-06-22 04:14:36.626898148 +0000 UTC m=+19.220504062 start trafficmonitor_1 | 2020-06-22T04:14:36.632952203Z DEBUG: cache.go:195: 2020-06-22T04:14:36.632622577Z: poll 1 2020-06-22 04:14:36.632620001 +0000 UTC m=+19.226225782 poller end trafficmonitor_1 | 2020-06-22T04:14:36.632997108Z DEBUG: cache.go:279: 2020-06-22T04:14:36.632759642Z: poll 1 2020-06-22 04:14:36.632756075 +0000 UTC m=+19.226361885 (format 'astats') handle start trafficmonitor_1 | 2020-06-22T04:14:36.633962619Z INFO: cache.go:49: 2020-06-22T04:14:36.633660422Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-22T04:14:36.634008384Z DEBUG: health.go:157: 2020-06-22T04:14:36.633865721Z: poll 1 2020-06-22 04:14:36.633864567 +0000 UTC m=+19.227470341 finish trafficmonitor_1 | 2020-06-22T04:14:37.287370190Z DEBUG: cache.go:183: 2020-06-22T04:14:37.286934086Z: poll 2 2020-06-22 04:14:37.286928321 +0000 UTC m=+19.880534125 start trafficmonitor_1 | 2020-06-22T04:14:37.288267123Z DEBUG: cache.go:195: 2020-06-22T04:14:37.28800429Z: poll 2 2020-06-22 04:14:37.288002224 +0000 UTC m=+19.881607991 poller end trafficmonitor_1 | 2020-06-22T04:14:37.288318349Z DEBUG: cache.go:279: 2020-06-22T04:14:37.288106728Z: poll 2 2020-06-22 04:14:37.288104637 +0000 UTC m=+19.881710410 (format 'astats') handle start trafficmonitor_1 | 2020-06-22T04:14:37.288340028Z WARNING: cache.go:290: 2020-06-22T04:14:37.288163255Z: mid handler given error 'id mid url http://172.22.0.8:80/_astats?application=system&inf.name=eth0 fetch error: Get "http://172.22.0.8:80/_astats?application=system&inf.name=eth0": dial tcp 172.22.0.8:80: connect: connection refused' trafficmonitor_1 | 2020-06-22T04:14:37.288528251Z DEBUG: health.go:157: 2020-06-22T04:14:37.288292577Z: poll 2 2020-06-22 04:14:37.288290271 +0000 UTC m=+19.881896136 finish trafficmonitor_1 | 2020-06-22T04:14:37.290849651Z panic: assignment to entry in nil map trafficmonitor_1 | 2020-06-22T04:14:37.290887681Z trafficmonitor_1 | 2020-06-22T04:14:37.290899365Z goroutine 28 [running]: trafficmonitor_1 | 2020-06-22T04:14:37.290909754Z github.com/apache/trafficcontrol/traffic_monitor/health.CalcAvailability(0xc0002b5320, 0x1, 0x1, 0xc17f55, 0x6, 0x0, 0xc0006b8480, 0xc000612d80, 0xc000612db0, 0xc0006b84b0, ...) trafficmonitor_1 | 2020-06-22T04:14:37.290920837Z /tmp/go/src/github.com/apache/trafficcontrol/traffic_monitor/health/cache.go:293 +0x1705 trafficmonitor_1 | 2020-06-22T04:14:37.290940174Z github.com/apache/trafficcontrol/traffic_monitor/manager.processHealthResult(0xc00069e000, 0xc00069c000, 0xc000698020, 0xc000694000, 0xc000698000, 0xc00029a0c0, 0xc0002a41a0, 0xc0006963c0, 0xc000698080, 0xc0002982d0, ...) trafficmonitor_1 | 2020-06-22T04:14:37.290983099Z /tmp/go/src/github.com/apache/trafficcontrol/traffic_monitor/manager/health.go:189 +0xa30 trafficmonitor_1 | 2020-06-22T04:14:37.290994284Z github.com/apache/trafficcontrol/traffic_monitor/manager.healthResultManagerListen.func1(0xc0002b5320, 0x1, 0x1) trafficmonitor_1 | 2020-06-22T04:14:37.291006886Z /tmp/go/src/github.com/apache/trafficcontrol/traffic_monitor/manager/health.go:90 +0x315 trafficmonitor_1 | 2020-06-22T04:14:37.291025663Z github.com/apache/trafficcontrol/traffic_monitor/manager.healthResultManagerListen(0xc00069e000, 0xc00069c000, 0xc000698020, 0xc000694000, 0xc000698000, 0xc00029a0c0, 0xc0002a41a0, 0xc00029a0c8, 0xc0002a41c0, 0xc0006963c0, ...) trafficmonitor_1 | 2020-06-22T04:14:37.291065197Z /tmp/go/src/github.com/apache/trafficcontrol/traffic_monitor/manager/health.go:127 +0x6ee trafficmonitor_1 | 2020-06-22T04:14:37.291077119Z created by github.com/apache/trafficcontrol/traffic_monitor/manager.StartHealthResultManager trafficmonitor_1 | 2020-06-22T04:14:37.291088439Z /tmp/go/src/github.com/apache/trafficcontrol/traffic_monitor/manager/health.go:54 +0x1e5 trafficmonitor_1 | 2020-06-22T04:14:37.294369123Z + cleanup trafficmonitor_1 | 2020-06-22T04:14:37.294420024Z + rm -f /tmp/tmp.okBhUfQzjf /tmp/tmp.rvdyDj1PkB ``` <details><summary>First 40 seconds of CDN-in-a-Box Traffic Monitor output the 50% of the time that it does not panic (click to expand):</summary> ```go trafficmonitor_1 | 2020-06-21T21:20:02.185302669Z + dlv --continue --listen=:2344 --accept-multiclient=true --headless=true --api-version=2 exec /opt/traffic_monitor/bin/traffic_monitor -- -opsCfg /opt/traffic_monitor/conf/traffic_ops.cfg -config /opt/traffic_monitor/conf/traffic_monitor.cfg trafficmonitor_1 | 2020-06-21T21:20:02.188927532Z API server listening at: [::]:2344 trafficmonitor_1 | 2020-06-21T21:20:02.531552994Z INFO: traffic_monitor.go:69: 2020-06-21T21:20:02.531363513Z: Starting with config {CacheHealthPollingInterval:6s CacheStatPollingInterval:6s MonitorConfigPollingInterval:15s HTTPTimeout:24h0m0s PeerPollingInterval:5s PeerOptimistic:true PeerOptimisticQuorumMin:0 MaxEvents:200 MaxStatHistory:5 MaxHealthHistory:5 HealthFlushInterval:20ms StatFlushInterval:20ms StatBufferInterval:0s LogLocationError:stdout LogLocationWarning:stdout LogLocationInfo:stdout LogLocationDebug:stdout LogLocationEvent:stdout ServeReadTimeout:24h0m0s ServeWriteTimeout:24h0m0s HealthToStatRatio:4 StaticFileDir:/opt/traffic_monitor/static/ CRConfigHistoryCount:20000 TrafficOpsMinRetryInterval:100ms TrafficOpsMaxRetryInterval:1m0s CRConfigBackupFile:/opt/traffic_monitor/crconfig.backup TMConfigBackupFile:/opt/traffic_monitor/tmconfig.backup TrafficOpsDiskRetryMax:2 CachePollingProtocol:both PeerPollingProtocol:both} trafficmonitor_1 | 2020-06-21T21:20:02.533139797Z INFO: srvhttp.go:95: 2020-06-21T21:20:02.533093347Z: Starting Web Server trafficmonitor_1 | 2020-06-21T21:20:02.533510982Z INFO: srvhttp.go:142: 2020-06-21T21:20:02.53346853Z: Web server listening on :80 trafficmonitor_1 | 2020-06-21T21:20:02.677913906Z INFO: monitorconfig.go:89: 2020-06-21T21:20:02.677801816Z: MonitorConfigPoller: received new opsConfig: {tmon jhdslvhdfsuklvfhsuvlhs https://trafficops.infra.ciab.test:443 true CDN-in-a-Box :80 } trafficmonitor_1 | 2020-06-21T21:20:02.677940906Z INFO: monitorconfig.go:92: 2020-06-21T21:20:02.677843924Z: MonitorConfigPoller: received new session: {0xc0000a8208 0xc000680008 {0xc0000a8210 0xc0000a4e20} {0xc0000a1000 0xc000682000 0xc0000a29f8 0xc00030a078 0xc000680000} /opt/traffic_monitor/crconfig.backup /opt/traffic_monitor/tmconfig.backup} trafficmonitor_1 | 2020-06-21T21:20:06.205869122Z ERROR: datareq.go:151: 2020-06-21T21:20:06.205727122Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-21T21:20:11.203346295Z ERROR: datareq.go:151: 2020-06-21T21:20:11.203038639Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-21T21:20:16.205709206Z ERROR: datareq.go:151: 2020-06-21T21:20:16.205413697Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-21T21:20:17.550877035Z INFO: monitorconfig.go:263: 2020-06-21T21:20:17.550666606Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:17.550929290Z INFO: monitorconfig.go:269: 2020-06-21T21:20:17.550722625Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:17.551680130Z INFO: monitorconfig.go:263: 2020-06-21T21:20:17.550897563Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:17.551726199Z INFO: monitorconfig.go:269: 2020-06-21T21:20:17.550933518Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:17.551742489Z INFO: monitorconfig.go:98: 2020-06-21T21:20:17.551046943Z: MonitorConfigPoller: received new interval: 1.94s trafficmonitor_1 | 2020-06-21T21:20:20.003280959Z ERROR: datareq.go:151: 2020-06-21T21:20:20.002964263Z: Request Error: /publish/CacheStats: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-21T21:20:20.003614016Z ERROR: datareq.go:151: 2020-06-21T21:20:20.003026281Z: Request Error: /publish/DsStats: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-21T21:20:21.210156275Z ERROR: datareq.go:151: 2020-06-21T21:20:21.208249338Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[] trafficmonitor_1 | 2020-06-21T21:20:21.736232969Z DEBUG: cache.go:183: 2020-06-21T21:20:21.735328408Z: poll 1 2020-06-21 21:20:21.735323802 +0000 UTC m=+19.209097518 start trafficmonitor_1 | 2020-06-21T21:20:21.741545518Z DEBUG: cache.go:195: 2020-06-21T21:20:21.741266475Z: poll 1 2020-06-21 21:20:21.741264646 +0000 UTC m=+19.215038353 poller end trafficmonitor_1 | 2020-06-21T21:20:21.741592144Z DEBUG: cache.go:279: 2020-06-21T21:20:21.741399726Z: poll 1 2020-06-21 21:20:21.741397539 +0000 UTC m=+19.215171238 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:21.742528049Z INFO: cache.go:49: 2020-06-21T21:20:21.742253304Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:21.742682716Z DEBUG: health.go:157: 2020-06-21T21:20:21.742480865Z: poll 1 2020-06-21 21:20:21.74247974 +0000 UTC m=+19.216253449 finish trafficmonitor_1 | 2020-06-21T21:20:22.395323056Z DEBUG: cache.go:183: 2020-06-21T21:20:22.395216632Z: poll 2 2020-06-21 21:20:22.395214503 +0000 UTC m=+19.868988134 start trafficmonitor_1 | 2020-06-21T21:20:22.396959343Z DEBUG: cache.go:195: 2020-06-21T21:20:22.396871926Z: poll 2 2020-06-21 21:20:22.396871423 +0000 UTC m=+19.870645051 poller end trafficmonitor_1 | 2020-06-21T21:20:22.396973594Z DEBUG: cache.go:279: 2020-06-21T21:20:22.39690157Z: poll 2 2020-06-21 21:20:22.396900953 +0000 UTC m=+19.870674584 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:22.397002597Z INFO: cache.go:49: 2020-06-21T21:20:22.396964045Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:22.397053122Z DEBUG: health.go:157: 2020-06-21T21:20:22.397012043Z: poll 2 2020-06-21 21:20:22.397011746 +0000 UTC m=+19.870785377 finish trafficmonitor_1 | 2020-06-21T21:20:25.844058238Z DEBUG: cache.go:183: 2020-06-21T21:20:25.8433244Z: poll 3 2020-06-21 21:20:25.843319124 +0000 UTC m=+23.317092836 start trafficmonitor_1 | 2020-06-21T21:20:25.850455305Z DEBUG: cache.go:195: 2020-06-21T21:20:25.85015222Z: poll 3 2020-06-21 21:20:25.850148605 +0000 UTC m=+23.323922339 poller end trafficmonitor_1 | 2020-06-21T21:20:25.850579607Z DEBUG: cache.go:279: 2020-06-21T21:20:25.850329596Z: poll 3 2020-06-21 21:20:25.850325365 +0000 UTC m=+23.324099140 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:25.854610937Z INFO: astats.go:134: 2020-06-21T21:20:25.85437032Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.854887915Z INFO: astats.go:134: 2020-06-21T21:20:25.854742172Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855141316Z INFO: astats.go:134: 2020-06-21T21:20:25.855016351Z: precomputing cache mid stat system.proc.loadavg value 1.79 2.02 1.61 1/2094 358 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855276575Z INFO: astats.go:134: 2020-06-21T21:20:25.855161473Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855357427Z INFO: astats.go:134: 2020-06-21T21:20:25.855249843Z: precomputing cache mid stat system.proc.net.dev value eth0: 550271 1130 0 0 0 0 0 0 176120 1299 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855477298Z INFO: astats.go:134: 2020-06-21T21:20:25.855310467Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855504520Z INFO: astats.go:134: 2020-06-21T21:20:25.855357779Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855533256Z INFO: astats.go:134: 2020-06-21T21:20:25.855444688Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855603029Z INFO: astats.go:134: 2020-06-21T21:20:25.855505504Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855617083Z INFO: astats.go:134: 2020-06-21T21:20:25.855591268Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:25.855648765Z INFO: stat.go:165: 2020-06-21T21:20:25.855615103Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:20:25.855661168Z INFO: stat.go:181: 2020-06-21T21:20:25.855631733Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:20:25.855726386Z INFO: stat.go:458: 2020-06-21T21:20:25.85569904Z: CreateStats took 8.759µs trafficmonitor_1 | 2020-06-21T21:20:25.855780683Z INFO: monitorconfig.go:263: 2020-06-21T21:20:25.855753376Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:25.855786353Z INFO: monitorconfig.go:269: 2020-06-21T21:20:25.855767502Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:25.855845807Z INFO: stat.go:200: 2020-06-21T21:20:25.855802156Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:20:25.855853128Z INFO: monitorconfig.go:263: 2020-06-21T21:20:25.855809286Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:25.855856231Z INFO: monitorconfig.go:269: 2020-06-21T21:20:25.855817873Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:26.209159911Z ERROR: datareq.go:151: 2020-06-21T21:20:26.208983894Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[edge:{} mid:{}] trafficmonitor_1 | 2020-06-21T21:20:27.369543487Z DEBUG: cache.go:183: 2020-06-21T21:20:27.368832691Z: poll 4 2020-06-21 21:20:27.368827402 +0000 UTC m=+24.842601116 start trafficmonitor_1 | 2020-06-21T21:20:27.375790754Z DEBUG: cache.go:195: 2020-06-21T21:20:27.375417471Z: poll 4 2020-06-21 21:20:27.375415039 +0000 UTC m=+24.849188756 poller end trafficmonitor_1 | 2020-06-21T21:20:27.375846270Z DEBUG: cache.go:279: 2020-06-21T21:20:27.375596504Z: poll 4 2020-06-21 21:20:27.37559381 +0000 UTC m=+24.849367524 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:27.380251852Z INFO: astats.go:134: 2020-06-21T21:20:27.379927617Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.380325593Z INFO: astats.go:134: 2020-06-21T21:20:27.380180805Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.380867255Z INFO: astats.go:134: 2020-06-21T21:20:27.380552445Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.380925206Z INFO: astats.go:134: 2020-06-21T21:20:27.380765318Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.382492642Z INFO: astats.go:134: 2020-06-21T21:20:27.382269295Z: precomputing cache edge stat system.proc.loadavg value 1.79 2.02 1.61 1/2096 360 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.383302774Z INFO: astats.go:134: 2020-06-21T21:20:27.383092036Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.384080277Z INFO: astats.go:134: 2020-06-21T21:20:27.38388911Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.384214595Z INFO: astats.go:134: 2020-06-21T21:20:27.384000189Z: precomputing cache edge stat system.proc.net.dev value eth0: 176721 436 0 0 0 0 0 0 56707 456 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.384733656Z INFO: astats.go:134: 2020-06-21T21:20:27.384508446Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.384791012Z INFO: astats.go:134: 2020-06-21T21:20:27.384658691Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:27.385112328Z INFO: stat.go:165: 2020-06-21T21:20:27.384923445Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:20:27.385146673Z INFO: stat.go:181: 2020-06-21T21:20:27.385033592Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:20:27.386041694Z INFO: monitorconfig.go:263: 2020-06-21T21:20:27.385804504Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:27.386113622Z INFO: monitorconfig.go:269: 2020-06-21T21:20:27.38594636Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:27.386443995Z INFO: monitorconfig.go:263: 2020-06-21T21:20:27.38630738Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:27.386494098Z INFO: monitorconfig.go:269: 2020-06-21T21:20:27.386370027Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:27.386521431Z INFO: stat.go:54: 2020-06-21T21:20:27.386379051Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:20:27.386591546Z 1592774427.386 host="demo1", type=Delivery Service, available=true, msg="available caches" trafficmonitor_1 | 2020-06-21T21:20:27.386648899Z 1592774427.387 host="demo1", type=DELIVERYSERVICE, available=true, msg="REPORTED - available" trafficmonitor_1 | 2020-06-21T21:20:27.386849423Z INFO: stat.go:458: 2020-06-21T21:20:27.386678307Z: CreateStats took 769.747µs trafficmonitor_1 | 2020-06-21T21:20:27.388119472Z INFO: stat.go:200: 2020-06-21T21:20:27.387995305Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:20:27.555861562Z DEBUG: cache.go:177: 2020-06-21T21:20:27.555326941Z: Intended Duration: 2.91s Actual Duration: 5.819997673s trafficmonitor_1 | 2020-06-21T21:20:27.555929879Z DEBUG: cache.go:183: 2020-06-21T21:20:27.555640962Z: poll 5 2020-06-21 21:20:27.555637006 +0000 UTC m=+25.029410744 start trafficmonitor_1 | 2020-06-21T21:20:27.560058897Z DEBUG: cache.go:195: 2020-06-21T21:20:27.559853152Z: poll 5 2020-06-21 21:20:27.559850606 +0000 UTC m=+25.033624350 poller end trafficmonitor_1 | 2020-06-21T21:20:27.560247099Z DEBUG: cache.go:279: 2020-06-21T21:20:27.560027076Z: poll 5 2020-06-21 21:20:27.560023621 +0000 UTC m=+25.033797364 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:27.560535235Z INFO: cache.go:49: 2020-06-21T21:20:27.560328599Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:27.560790897Z DEBUG: health.go:157: 2020-06-21T21:20:27.560644753Z: poll 5 2020-06-21 21:20:27.560643101 +0000 UTC m=+25.034416813 finish trafficmonitor_1 | 2020-06-21T21:20:28.215598857Z DEBUG: cache.go:177: 2020-06-21T21:20:28.215263463Z: Intended Duration: 2.91s Actual Duration: 5.820046167s trafficmonitor_1 | 2020-06-21T21:20:28.215687124Z DEBUG: cache.go:183: 2020-06-21T21:20:28.215361956Z: poll 6 2020-06-21 21:20:28.21535961 +0000 UTC m=+25.689133374 start trafficmonitor_1 | 2020-06-21T21:20:28.220091024Z DEBUG: cache.go:195: 2020-06-21T21:20:28.219760615Z: poll 6 2020-06-21 21:20:28.219757974 +0000 UTC m=+25.693531699 poller end trafficmonitor_1 | 2020-06-21T21:20:28.220189053Z DEBUG: cache.go:279: 2020-06-21T21:20:28.219883659Z: poll 6 2020-06-21 21:20:28.219880767 +0000 UTC m=+25.693654483 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:28.220338007Z INFO: cache.go:49: 2020-06-21T21:20:28.220135671Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:28.220499524Z DEBUG: health.go:157: 2020-06-21T21:20:28.220331885Z: poll 6 2020-06-21 21:20:28.220330544 +0000 UTC m=+25.694104252 finish trafficmonitor_1 | 2020-06-21T21:20:30.001568956Z ERROR: datareq.go:151: 2020-06-21T21:20:30.001238976Z: Request Error: /publish/DsStats: service still starting, some caches unpolled: map[edge:{} mid:{}] trafficmonitor_1 | 2020-06-21T21:20:30.001882798Z ERROR: datareq.go:151: 2020-06-21T21:20:30.001632753Z: Request Error: /publish/CacheStats: service still starting, some caches unpolled: map[edge:{} mid:{}] trafficmonitor_1 | 2020-06-21T21:20:31.211701436Z ERROR: datareq.go:151: 2020-06-21T21:20:31.211453261Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[edge:{} mid:{}] trafficmonitor_1 | 2020-06-21T21:20:33.375681236Z DEBUG: cache.go:177: 2020-06-21T21:20:33.3753664Z: Intended Duration: 2.91s Actual Duration: 5.819727184s trafficmonitor_1 | 2020-06-21T21:20:33.375749488Z DEBUG: cache.go:183: 2020-06-21T21:20:33.375464055Z: poll 7 2020-06-21 21:20:33.375460548 +0000 UTC m=+30.849234259 start trafficmonitor_1 | 2020-06-21T21:20:33.379898494Z DEBUG: cache.go:195: 2020-06-21T21:20:33.379607642Z: poll 7 2020-06-21 21:20:33.379605656 +0000 UTC m=+30.853379367 poller end trafficmonitor_1 | 2020-06-21T21:20:33.379962649Z DEBUG: cache.go:279: 2020-06-21T21:20:33.379722453Z: poll 7 2020-06-21 21:20:33.379720044 +0000 UTC m=+30.853493773 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:33.380165437Z INFO: cache.go:49: 2020-06-21T21:20:33.379943383Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:33.380429710Z DEBUG: health.go:157: 2020-06-21T21:20:33.380233406Z: poll 7 2020-06-21 21:20:33.380231437 +0000 UTC m=+30.854005164 finish trafficmonitor_1 | 2020-06-21T21:20:34.035611492Z DEBUG: cache.go:177: 2020-06-21T21:20:34.035343447Z: Intended Duration: 2.91s Actual Duration: 5.819981618s trafficmonitor_1 | 2020-06-21T21:20:34.035678949Z DEBUG: cache.go:183: 2020-06-21T21:20:34.035437731Z: poll 8 2020-06-21 21:20:34.035435491 +0000 UTC m=+31.509209214 start trafficmonitor_1 | 2020-06-21T21:20:34.040535319Z DEBUG: cache.go:195: 2020-06-21T21:20:34.040298473Z: poll 8 2020-06-21 21:20:34.040295927 +0000 UTC m=+31.514069668 poller end trafficmonitor_1 | 2020-06-21T21:20:34.040597500Z DEBUG: cache.go:279: 2020-06-21T21:20:34.040433533Z: poll 8 2020-06-21 21:20:34.040430482 +0000 UTC m=+31.514204202 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:34.040850886Z INFO: cache.go:49: 2020-06-21T21:20:34.040668462Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:34.041158442Z DEBUG: health.go:157: 2020-06-21T21:20:34.040958817Z: poll 8 2020-06-21 21:20:34.040957564 +0000 UTC m=+31.514731272 finish trafficmonitor_1 | 2020-06-21T21:20:36.212239159Z ERROR: datareq.go:151: 2020-06-21T21:20:36.212100735Z: Request Error: /publish/CrConfig: service still starting, some caches unpolled: map[edge:{} mid:{}] trafficmonitor_1 | 2020-06-21T21:20:37.483899822Z DEBUG: cache.go:177: 2020-06-21T21:20:37.483594387Z: Intended Duration: 5.82s Actual Duration: 11.63996559s trafficmonitor_1 | 2020-06-21T21:20:37.483984345Z DEBUG: cache.go:183: 2020-06-21T21:20:37.483696868Z: poll 9 2020-06-21 21:20:37.483694426 +0000 UTC m=+34.957468151 start trafficmonitor_1 | 2020-06-21T21:20:37.489740322Z DEBUG: cache.go:195: 2020-06-21T21:20:37.489496018Z: poll 9 2020-06-21 21:20:37.489493794 +0000 UTC m=+34.963267513 poller end trafficmonitor_1 | 2020-06-21T21:20:37.489793904Z DEBUG: cache.go:279: 2020-06-21T21:20:37.489607162Z: poll 9 2020-06-21 21:20:37.489604478 +0000 UTC m=+34.963378197 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:37.492705647Z INFO: astats.go:134: 2020-06-21T21:20:37.492481753Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.492758895Z INFO: astats.go:134: 2020-06-21T21:20:37.492604137Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493040331Z INFO: astats.go:134: 2020-06-21T21:20:37.492868965Z: precomputing cache mid stat system.proc.net.dev value eth0: 551360 1139 0 0 0 0 0 0 214317 1308 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493098601Z INFO: astats.go:134: 2020-06-21T21:20:37.492934011Z: precomputing cache mid stat system.proc.loadavg value 1.59 1.97 1.60 3/2098 359 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493121323Z INFO: astats.go:134: 2020-06-21T21:20:37.493007423Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493417984Z INFO: astats.go:134: 2020-06-21T21:20:37.493237033Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493662875Z INFO: astats.go:134: 2020-06-21T21:20:37.493513763Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493718366Z INFO: astats.go:134: 2020-06-21T21:20:37.493584288Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493755303Z INFO: astats.go:134: 2020-06-21T21:20:37.493634714Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.493939686Z INFO: astats.go:134: 2020-06-21T21:20:37.493792372Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:37.494167345Z INFO: stat.go:165: 2020-06-21T21:20:37.493978395Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:20:37.494211711Z INFO: stat.go:181: 2020-06-21T21:20:37.494051017Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:20:37.494317920Z INFO: cache.go:49: 2020-06-21T21:20:37.494147533Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:37.494700491Z INFO: monitorconfig.go:263: 2020-06-21T21:20:37.494501712Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:37.494757554Z INFO: monitorconfig.go:269: 2020-06-21T21:20:37.494569891Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:37.494825230Z INFO: monitorconfig.go:263: 2020-06-21T21:20:37.494703416Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:37.494858257Z INFO: monitorconfig.go:269: 2020-06-21T21:20:37.494739046Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:37.506904792Z INFO: stat.go:54: 2020-06-21T21:20:37.506627012Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:20:37.506973035Z 1592774437.507 host="demo1", type=Delivery Service, available=false, msg="no available caches" trafficmonitor_1 | 2020-06-21T21:20:37.507068483Z INFO: stat.go:458: 2020-06-21T21:20:37.50690207Z: CreateStats took 298.062µs trafficmonitor_1 | 2020-06-21T21:20:37.507551102Z DEBUG: polledcaches.go:170: 2020-06-21T21:20:37.507363833Z: polled edge (byte change timed out) trafficmonitor_1 | 2020-06-21T21:20:37.507598944Z DEBUG: polledcaches.go:160: 2020-06-21T21:20:37.507442689Z: polled mid trafficmonitor_1 | 2020-06-21T21:20:37.507621997Z INFO: polledcaches.go:184: 2020-06-21T21:20:37.507478286Z: all caches polled, ready to serve! trafficmonitor_1 | 2020-06-21T21:20:37.507695815Z INFO: stat.go:200: 2020-06-21T21:20:37.507512227Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:20:39.009387823Z DEBUG: cache.go:177: 2020-06-21T21:20:39.008791131Z: Intended Duration: 5.82s Actual Duration: 11.639963322s trafficmonitor_1 | 2020-06-21T21:20:39.009474061Z DEBUG: cache.go:183: 2020-06-21T21:20:39.009144063Z: poll 10 2020-06-21 21:20:39.009141475 +0000 UTC m=+36.482915191 start trafficmonitor_1 | 2020-06-21T21:20:39.014292548Z DEBUG: cache.go:195: 2020-06-21T21:20:39.014073579Z: poll 10 2020-06-21 21:20:39.014071547 +0000 UTC m=+36.487845259 poller end trafficmonitor_1 | 2020-06-21T21:20:39.014394677Z DEBUG: cache.go:279: 2020-06-21T21:20:39.014196487Z: poll 10 2020-06-21 21:20:39.014193188 +0000 UTC m=+36.487966991 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:39.017417457Z INFO: astats.go:134: 2020-06-21T21:20:39.017131614Z: precomputing cache edge stat system.proc.net.dev value eth0: 177883 446 0 0 0 0 0 0 94928 465 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.017472666Z INFO: astats.go:134: 2020-06-21T21:20:39.017227843Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.017489226Z INFO: astats.go:134: 2020-06-21T21:20:39.017299123Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.017513263Z INFO: astats.go:134: 2020-06-21T21:20:39.017365273Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.017549036Z INFO: astats.go:134: 2020-06-21T21:20:39.017482169Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.017892903Z INFO: astats.go:134: 2020-06-21T21:20:39.017712337Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.018149184Z INFO: astats.go:134: 2020-06-21T21:20:39.017965665Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.018199960Z INFO: astats.go:134: 2020-06-21T21:20:39.018099381Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.018356114Z INFO: astats.go:134: 2020-06-21T21:20:39.018219915Z: precomputing cache edge stat system.proc.loadavg value 1.59 1.97 1.60 2/2099 361 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.018634598Z INFO: astats.go:134: 2020-06-21T21:20:39.01838635Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:39.018699421Z INFO: stat.go:165: 2020-06-21T21:20:39.018529788Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:20:39.018746899Z INFO: stat.go:181: 2020-06-21T21:20:39.018580339Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:20:39.018876203Z INFO: cache.go:49: 2020-06-21T21:20:39.018716151Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:39.019302985Z INFO: monitorconfig.go:263: 2020-06-21T21:20:39.019118268Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:39.019392427Z INFO: monitorconfig.go:269: 2020-06-21T21:20:39.019184946Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:39.019473428Z INFO: monitorconfig.go:263: 2020-06-21T21:20:39.019333843Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:39.019595893Z INFO: monitorconfig.go:269: 2020-06-21T21:20:39.019381175Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:39.023709112Z INFO: stat.go:54: 2020-06-21T21:20:39.023467232Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:20:39.023759416Z 1592774439.024 host="demo1", type=Delivery Service, available=true, msg="available caches" trafficmonitor_1 | 2020-06-21T21:20:39.023775884Z 1592774439.024 host="demo1", type=DELIVERYSERVICE, available=true, msg="REPORTED - available" trafficmonitor_1 | 2020-06-21T21:20:39.023837883Z INFO: stat.go:458: 2020-06-21T21:20:39.023665187Z: CreateStats took 211.165µs trafficmonitor_1 | 2020-06-21T21:20:39.024012799Z INFO: stat.go:200: 2020-06-21T21:20:39.02388732Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:20:39.195680681Z DEBUG: cache.go:177: 2020-06-21T21:20:39.195355554Z: Intended Duration: 2.91s Actual Duration: 5.819892877s trafficmonitor_1 | 2020-06-21T21:20:39.196042219Z DEBUG: cache.go:183: 2020-06-21T21:20:39.195809671Z: poll 11 2020-06-21 21:20:39.195805122 +0000 UTC m=+36.669578876 start trafficmonitor_1 | 2020-06-21T21:20:39.201247403Z DEBUG: cache.go:195: 2020-06-21T21:20:39.200991707Z: poll 11 2020-06-21 21:20:39.200989014 +0000 UTC m=+36.674762765 poller end trafficmonitor_1 | 2020-06-21T21:20:39.201321170Z DEBUG: cache.go:279: 2020-06-21T21:20:39.201156129Z: poll 11 2020-06-21 21:20:39.201152816 +0000 UTC m=+36.674926565 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:39.201567107Z INFO: cache.go:49: 2020-06-21T21:20:39.201380334Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:39.201753253Z DEBUG: health.go:157: 2020-06-21T21:20:39.201625459Z: poll 11 2020-06-21 21:20:39.201624169 +0000 UTC m=+36.675397880 finish trafficmonitor_1 | 2020-06-21T21:20:39.855683532Z DEBUG: cache.go:177: 2020-06-21T21:20:39.855347165Z: Intended Duration: 2.91s Actual Duration: 5.81990866s trafficmonitor_1 | 2020-06-21T21:20:39.855775868Z DEBUG: cache.go:183: 2020-06-21T21:20:39.855445651Z: poll 12 2020-06-21 21:20:39.855443066 +0000 UTC m=+37.329216779 start trafficmonitor_1 | 2020-06-21T21:20:39.860206694Z DEBUG: cache.go:195: 2020-06-21T21:20:39.859908554Z: poll 12 2020-06-21 21:20:39.859906058 +0000 UTC m=+37.333679786 poller end trafficmonitor_1 | 2020-06-21T21:20:39.860386364Z DEBUG: cache.go:279: 2020-06-21T21:20:39.860064745Z: poll 12 2020-06-21 21:20:39.860060971 +0000 UTC m=+37.333834726 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:39.860577338Z INFO: cache.go:49: 2020-06-21T21:20:39.860405311Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:39.860881156Z DEBUG: health.go:157: 2020-06-21T21:20:39.860651851Z: poll 12 2020-06-21 21:20:39.860649707 +0000 UTC m=+37.334423446 finish trafficmonitor_1 | 2020-06-21T21:20:45.015599835Z DEBUG: cache.go:177: 2020-06-21T21:20:45.015336503Z: Intended Duration: 2.91s Actual Duration: 5.81952981s trafficmonitor_1 | 2020-06-21T21:20:45.015665699Z DEBUG: cache.go:183: 2020-06-21T21:20:45.015430852Z: poll 13 2020-06-21 21:20:45.0154283 +0000 UTC m=+42.489201999 start trafficmonitor_1 | 2020-06-21T21:20:45.019886329Z DEBUG: cache.go:195: 2020-06-21T21:20:45.019659137Z: poll 13 2020-06-21 21:20:45.01965722 +0000 UTC m=+42.493430931 poller end trafficmonitor_1 | 2020-06-21T21:20:45.019954100Z DEBUG: cache.go:279: 2020-06-21T21:20:45.019779138Z: poll 13 2020-06-21 21:20:45.019776437 +0000 UTC m=+42.493550154 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:45.020268363Z INFO: cache.go:49: 2020-06-21T21:20:45.020008413Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:45.020492878Z DEBUG: health.go:157: 2020-06-21T21:20:45.020304381Z: poll 13 2020-06-21 21:20:45.020303019 +0000 UTC m=+42.494076723 finish trafficmonitor_1 | 2020-06-21T21:20:45.675417968Z DEBUG: cache.go:177: 2020-06-21T21:20:45.675254335Z: Intended Duration: 2.91s Actual Duration: 5.81981146s trafficmonitor_1 | 2020-06-21T21:20:45.675439794Z DEBUG: cache.go:183: 2020-06-21T21:20:45.675379258Z: poll 14 2020-06-21 21:20:45.67537807 +0000 UTC m=+43.149151698 start trafficmonitor_1 | 2020-06-21T21:20:45.676561909Z DEBUG: cache.go:195: 2020-06-21T21:20:45.676475946Z: poll 14 2020-06-21 21:20:45.676475384 +0000 UTC m=+43.150249018 poller end trafficmonitor_1 | 2020-06-21T21:20:45.676577958Z DEBUG: cache.go:279: 2020-06-21T21:20:45.676510179Z: poll 14 2020-06-21 21:20:45.676509463 +0000 UTC m=+43.150283094 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:45.676627764Z INFO: cache.go:49: 2020-06-21T21:20:45.67658459Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:45.676695729Z DEBUG: health.go:157: 2020-06-21T21:20:45.676644579Z: poll 14 2020-06-21 21:20:45.676644267 +0000 UTC m=+43.150417895 finish trafficmonitor_1 | 2020-06-21T21:20:49.123909958Z DEBUG: cache.go:177: 2020-06-21T21:20:49.123299354Z: Intended Duration: 5.82s Actual Duration: 11.639603141s trafficmonitor_1 | 2020-06-21T21:20:49.123992499Z DEBUG: cache.go:183: 2020-06-21T21:20:49.123730943Z: poll 15 2020-06-21 21:20:49.123728065 +0000 UTC m=+46.597501790 start trafficmonitor_1 | 2020-06-21T21:20:49.128988721Z DEBUG: cache.go:195: 2020-06-21T21:20:49.128726553Z: poll 15 2020-06-21 21:20:49.128724387 +0000 UTC m=+46.602498107 poller end trafficmonitor_1 | 2020-06-21T21:20:49.129096572Z DEBUG: cache.go:279: 2020-06-21T21:20:49.128843099Z: poll 15 2020-06-21 21:20:49.128840696 +0000 UTC m=+46.602614409 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:49.131703517Z INFO: astats.go:134: 2020-06-21T21:20:49.131465116Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.131770179Z INFO: astats.go:134: 2020-06-21T21:20:49.131534328Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.131912781Z INFO: astats.go:134: 2020-06-21T21:20:49.131765307Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132225261Z INFO: astats.go:134: 2020-06-21T21:20:49.132035605Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132296440Z INFO: astats.go:134: 2020-06-21T21:20:49.132138513Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132325382Z INFO: astats.go:134: 2020-06-21T21:20:49.132201281Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132346941Z INFO: astats.go:134: 2020-06-21T21:20:49.132231553Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132509951Z INFO: astats.go:134: 2020-06-21T21:20:49.132320477Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132556701Z INFO: astats.go:134: 2020-06-21T21:20:49.132376825Z: precomputing cache mid stat system.proc.loadavg value 1.34 1.91 1.58 2/2108 360 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.132674526Z INFO: astats.go:134: 2020-06-21T21:20:49.132451808Z: precomputing cache mid stat system.proc.net.dev value eth0: 552491 1149 0 0 0 0 0 0 252494 1316 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:49.133205926Z INFO: stat.go:165: 2020-06-21T21:20:49.132990543Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:20:49.133252002Z INFO: stat.go:181: 2020-06-21T21:20:49.133043621Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:20:49.133274034Z INFO: cache.go:49: 2020-06-21T21:20:49.133115516Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:49.135563159Z INFO: monitorconfig.go:263: 2020-06-21T21:20:49.133509105Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:49.135623793Z INFO: monitorconfig.go:269: 2020-06-21T21:20:49.133596673Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:49.135652172Z INFO: monitorconfig.go:263: 2020-06-21T21:20:49.133767594Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:49.135676014Z INFO: monitorconfig.go:269: 2020-06-21T21:20:49.133812764Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:49.135696752Z ERROR: stat.go:292: 2020-06-21T21:20:49.135277764Z: Adding result from mid: some stats could not be added: cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:20:49.135723022Z INFO: stat.go:54: 2020-06-21T21:20:49.135379324Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:20:49.135747373Z INFO: stat.go:458: 2020-06-21T21:20:49.135478594Z: CreateStats took 115.666µs trafficmonitor_1 | 2020-06-21T21:20:49.135911912Z INFO: stat.go:200: 2020-06-21T21:20:49.135746285Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:20:50.649117772Z DEBUG: cache.go:177: 2020-06-21T21:20:50.648785321Z: Intended Duration: 5.82s Actual Duration: 11.639641731s trafficmonitor_1 | 2020-06-21T21:20:50.649200811Z DEBUG: cache.go:183: 2020-06-21T21:20:50.648876525Z: poll 16 2020-06-21 21:20:50.648872882 +0000 UTC m=+48.122646591 start trafficmonitor_1 | 2020-06-21T21:20:50.654238830Z DEBUG: cache.go:195: 2020-06-21T21:20:50.653954915Z: poll 16 2020-06-21 21:20:50.653952861 +0000 UTC m=+48.127726560 poller end trafficmonitor_1 | 2020-06-21T21:20:50.654309803Z DEBUG: cache.go:279: 2020-06-21T21:20:50.654076837Z: poll 16 2020-06-21 21:20:50.654073069 +0000 UTC m=+48.127846794 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:50.656949412Z INFO: astats.go:134: 2020-06-21T21:20:50.656696877Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657182572Z INFO: astats.go:134: 2020-06-21T21:20:50.656999114Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657234552Z INFO: astats.go:134: 2020-06-21T21:20:50.657066185Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657262460Z INFO: astats.go:134: 2020-06-21T21:20:50.657133221Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657417933Z INFO: astats.go:134: 2020-06-21T21:20:50.657290063Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657520179Z INFO: astats.go:134: 2020-06-21T21:20:50.657381881Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657691421Z INFO: astats.go:134: 2020-06-21T21:20:50.657511289Z: precomputing cache edge stat system.proc.net.dev value eth0: 179083 457 0 0 0 0 0 0 133175 474 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657873494Z INFO: astats.go:134: 2020-06-21T21:20:50.65771117Z: precomputing cache edge stat system.proc.loadavg value 1.31 1.89 1.58 2/2112 363 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.657942326Z INFO: astats.go:134: 2020-06-21T21:20:50.657801764Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.658059625Z INFO: astats.go:134: 2020-06-21T21:20:50.657927758Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:20:50.658279577Z INFO: stat.go:165: 2020-06-21T21:20:50.658128093Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:20:50.658309290Z INFO: stat.go:181: 2020-06-21T21:20:50.658187299Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:20:50.658464397Z INFO: cache.go:49: 2020-06-21T21:20:50.658330396Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:50.658855798Z INFO: monitorconfig.go:263: 2020-06-21T21:20:50.658693207Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:50.658906762Z INFO: monitorconfig.go:269: 2020-06-21T21:20:50.658753769Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:50.659047896Z INFO: monitorconfig.go:263: 2020-06-21T21:20:50.658920806Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:20:50.659099362Z INFO: monitorconfig.go:269: 2020-06-21T21:20:50.65896915Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:20:50.660557336Z ERROR: stat.go:292: 2020-06-21T21:20:50.660384001Z: Adding result from edge: some stats could not be added: cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:20:50.660610567Z INFO: stat.go:54: 2020-06-21T21:20:50.660460582Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:20:50.660631993Z INFO: stat.go:458: 2020-06-21T21:20:50.660511024Z: CreateStats took 57.79µs trafficmonitor_1 | 2020-06-21T21:20:50.660809267Z INFO: stat.go:200: 2020-06-21T21:20:50.660688759Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:20:50.835668799Z DEBUG: cache.go:177: 2020-06-21T21:20:50.835340552Z: Intended Duration: 2.91s Actual Duration: 5.819909732s trafficmonitor_1 | 2020-06-21T21:20:50.835721881Z DEBUG: cache.go:183: 2020-06-21T21:20:50.835425241Z: poll 17 2020-06-21 21:20:50.835423102 +0000 UTC m=+48.309196804 start trafficmonitor_1 | 2020-06-21T21:20:50.839293935Z DEBUG: cache.go:195: 2020-06-21T21:20:50.839026023Z: poll 17 2020-06-21 21:20:50.839023839 +0000 UTC m=+48.312797562 poller end trafficmonitor_1 | 2020-06-21T21:20:50.839347820Z DEBUG: cache.go:279: 2020-06-21T21:20:50.839163897Z: poll 17 2020-06-21 21:20:50.839160815 +0000 UTC m=+48.312934523 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:50.839603107Z INFO: cache.go:49: 2020-06-21T21:20:50.839356773Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:50.839670761Z DEBUG: health.go:157: 2020-06-21T21:20:50.839555867Z: poll 17 2020-06-21 21:20:50.839554626 +0000 UTC m=+48.313328327 finish trafficmonitor_1 | 2020-06-21T21:20:51.495764544Z DEBUG: cache.go:177: 2020-06-21T21:20:51.495216524Z: Intended Duration: 2.91s Actual Duration: 5.819831235s trafficmonitor_1 | 2020-06-21T21:20:51.495837564Z DEBUG: cache.go:183: 2020-06-21T21:20:51.495548475Z: poll 18 2020-06-21 21:20:51.495544805 +0000 UTC m=+48.969318525 start trafficmonitor_1 | 2020-06-21T21:20:51.500846520Z DEBUG: cache.go:195: 2020-06-21T21:20:51.500012768Z: poll 18 2020-06-21 21:20:51.500010555 +0000 UTC m=+48.973784273 poller end trafficmonitor_1 | 2020-06-21T21:20:51.500907070Z DEBUG: cache.go:279: 2020-06-21T21:20:51.500178343Z: poll 18 2020-06-21 21:20:51.50017532 +0000 UTC m=+48.973949053 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:51.500930229Z INFO: cache.go:49: 2020-06-21T21:20:51.500421488Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:51.500948113Z DEBUG: health.go:157: 2020-06-21T21:20:51.500593309Z: poll 18 2020-06-21 21:20:51.500592031 +0000 UTC m=+48.974365742 finish trafficmonitor_1 | 2020-06-21T21:20:56.655889090Z DEBUG: cache.go:177: 2020-06-21T21:20:56.655381329Z: Intended Duration: 2.91s Actual Duration: 5.819955628s trafficmonitor_1 | 2020-06-21T21:20:56.655972680Z DEBUG: cache.go:183: 2020-06-21T21:20:56.655695149Z: poll 19 2020-06-21 21:20:56.655691502 +0000 UTC m=+54.129465256 start trafficmonitor_1 | 2020-06-21T21:20:56.660264380Z DEBUG: cache.go:195: 2020-06-21T21:20:56.659937639Z: poll 19 2020-06-21 21:20:56.659935373 +0000 UTC m=+54.133709107 poller end trafficmonitor_1 | 2020-06-21T21:20:56.660361422Z DEBUG: cache.go:279: 2020-06-21T21:20:56.660070289Z: poll 19 2020-06-21 21:20:56.660066982 +0000 UTC m=+54.133840746 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:56.660573889Z INFO: cache.go:49: 2020-06-21T21:20:56.660373718Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:56.660868308Z DEBUG: health.go:157: 2020-06-21T21:20:56.660644118Z: poll 19 2020-06-21 21:20:56.660642274 +0000 UTC m=+54.134416011 finish trafficmonitor_1 | 2020-06-21T21:20:57.315882437Z DEBUG: cache.go:177: 2020-06-21T21:20:57.315362958Z: Intended Duration: 2.91s Actual Duration: 5.819812075s trafficmonitor_1 | 2020-06-21T21:20:57.315955613Z DEBUG: cache.go:183: 2020-06-21T21:20:57.315699607Z: poll 20 2020-06-21 21:20:57.315692326 +0000 UTC m=+54.789466039 start trafficmonitor_1 | 2020-06-21T21:20:57.321479477Z DEBUG: cache.go:195: 2020-06-21T21:20:57.321188023Z: poll 20 2020-06-21 21:20:57.321185894 +0000 UTC m=+54.794959613 poller end trafficmonitor_1 | 2020-06-21T21:20:57.321565788Z DEBUG: cache.go:279: 2020-06-21T21:20:57.32130266Z: poll 20 2020-06-21 21:20:57.321300344 +0000 UTC m=+54.795074069 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:20:57.321861489Z INFO: cache.go:49: 2020-06-21T21:20:57.321647228Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:20:57.322105518Z DEBUG: health.go:157: 2020-06-21T21:20:57.321881785Z: poll 20 2020-06-21 21:20:57.321878018 +0000 UTC m=+54.795652831 finish trafficmonitor_1 | 2020-06-21T21:21:00.763576627Z DEBUG: cache.go:177: 2020-06-21T21:21:00.763268967Z: Intended Duration: 5.82s Actual Duration: 11.639535212s trafficmonitor_1 | 2020-06-21T21:21:00.763652948Z DEBUG: cache.go:183: 2020-06-21T21:21:00.763377213Z: poll 21 2020-06-21 21:21:00.763372473 +0000 UTC m=+58.237146222 start trafficmonitor_1 | 2020-06-21T21:21:00.768296984Z DEBUG: cache.go:195: 2020-06-21T21:21:00.768010186Z: poll 21 2020-06-21 21:21:00.768008104 +0000 UTC m=+58.241781817 poller end trafficmonitor_1 | 2020-06-21T21:21:00.768351476Z DEBUG: cache.go:279: 2020-06-21T21:21:00.768124034Z: poll 21 2020-06-21 21:21:00.768121912 +0000 UTC m=+58.241895624 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:00.770648367Z INFO: astats.go:134: 2020-06-21T21:21:00.77037373Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.770712065Z INFO: astats.go:134: 2020-06-21T21:21:00.770487535Z: precomputing cache mid stat system.proc.loadavg value 1.11 1.83 1.56 2/2212 361 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.770735763Z INFO: astats.go:134: 2020-06-21T21:21:00.77054784Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.770945009Z INFO: astats.go:134: 2020-06-21T21:21:00.770746777Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.770997579Z INFO: astats.go:134: 2020-06-21T21:21:00.770849419Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.771023176Z INFO: astats.go:134: 2020-06-21T21:21:00.770888922Z: precomputing cache mid stat system.proc.net.dev value eth0: 553580 1158 0 0 0 0 0 0 290618 1323 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.771072763Z INFO: astats.go:134: 2020-06-21T21:21:00.770944196Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.771141386Z INFO: astats.go:134: 2020-06-21T21:21:00.771034126Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.771231068Z INFO: astats.go:134: 2020-06-21T21:21:00.771104677Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.771467799Z INFO: astats.go:134: 2020-06-21T21:21:00.771311985Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:00.771613750Z INFO: stat.go:165: 2020-06-21T21:21:00.771482253Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:00.771666970Z INFO: stat.go:181: 2020-06-21T21:21:00.771538396Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:00.771841913Z INFO: cache.go:49: 2020-06-21T21:21:00.771703597Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:00.772207270Z INFO: monitorconfig.go:263: 2020-06-21T21:21:00.772036615Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:00.772244864Z INFO: monitorconfig.go:269: 2020-06-21T21:21:00.772116283Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:00.772386703Z INFO: monitorconfig.go:263: 2020-06-21T21:21:00.77227107Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:00.772442541Z INFO: monitorconfig.go:269: 2020-06-21T21:21:00.772311684Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:00.773996376Z ERROR: stat.go:292: 2020-06-21T21:21:00.773768318Z: Adding result from mid: some stats could not be added: cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:00.774051680Z INFO: stat.go:54: 2020-06-21T21:21:00.773858095Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:00.774067744Z INFO: stat.go:458: 2020-06-21T21:21:00.773900809Z: CreateStats took 56.326µs trafficmonitor_1 | 2020-06-21T21:21:00.774193094Z INFO: stat.go:200: 2020-06-21T21:21:00.774052968Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:02.288767295Z DEBUG: cache.go:177: 2020-06-21T21:21:02.288696873Z: Intended Duration: 5.82s Actual Duration: 11.639822468s trafficmonitor_1 | 2020-06-21T21:21:02.288784475Z DEBUG: cache.go:183: 2020-06-21T21:21:02.288726974Z: poll 22 2020-06-21 21:21:02.288726357 +0000 UTC m=+59.762499980 start trafficmonitor_1 | 2020-06-21T21:21:02.289907369Z DEBUG: cache.go:195: 2020-06-21T21:21:02.289854645Z: poll 22 2020-06-21 21:21:02.289854151 +0000 UTC m=+59.763627775 poller end trafficmonitor_1 | 2020-06-21T21:21:02.289928998Z DEBUG: cache.go:279: 2020-06-21T21:21:02.289890186Z: poll 22 2020-06-21 21:21:02.289889091 +0000 UTC m=+59.763662724 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:02.290445792Z INFO: astats.go:134: 2020-06-21T21:21:02.290409575Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290460039Z INFO: astats.go:134: 2020-06-21T21:21:02.290426631Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290469662Z INFO: astats.go:134: 2020-06-21T21:21:02.290443971Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290483655Z INFO: astats.go:134: 2020-06-21T21:21:02.290461565Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290526517Z INFO: astats.go:134: 2020-06-21T21:21:02.290503403Z: precomputing cache edge stat system.proc.loadavg value 1.11 1.83 1.56 5/2222 386 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290550224Z INFO: astats.go:134: 2020-06-21T21:21:02.290519972Z: precomputing cache edge stat system.proc.net.dev value eth0: 203824 516 0 0 0 0 0 0 179336 539 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290641420Z INFO: astats.go:134: 2020-06-21T21:21:02.290600668Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290662518Z INFO: astats.go:134: 2020-06-21T21:21:02.290633608Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290691984Z INFO: astats.go:134: 2020-06-21T21:21:02.290658512Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290783505Z INFO: astats.go:134: 2020-06-21T21:21:02.290692206Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:02.290796316Z INFO: stat.go:165: 2020-06-21T21:21:02.290727984Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:02.290801565Z INFO: stat.go:181: 2020-06-21T21:21:02.290737964Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:02.290812950Z INFO: cache.go:49: 2020-06-21T21:21:02.290773797Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:02.290959252Z INFO: monitorconfig.go:263: 2020-06-21T21:21:02.290925813Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:02.290968269Z INFO: monitorconfig.go:269: 2020-06-21T21:21:02.290944809Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:02.291011978Z INFO: monitorconfig.go:263: 2020-06-21T21:21:02.290988607Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:02.291023301Z INFO: monitorconfig.go:269: 2020-06-21T21:21:02.290999685Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:02.291312009Z ERROR: stat.go:292: 2020-06-21T21:21:02.291284236Z: Adding result from edge: some stats could not be added: cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:02.291323819Z INFO: stat.go:54: 2020-06-21T21:21:02.291303053Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:02.291334680Z INFO: stat.go:458: 2020-06-21T21:21:02.291316699Z: CreateStats took 15.873µs trafficmonitor_1 | 2020-06-21T21:21:02.291385976Z INFO: stat.go:200: 2020-06-21T21:21:02.29136444Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:02.475368508Z DEBUG: cache.go:177: 2020-06-21T21:21:02.475228872Z: Intended Duration: 2.91s Actual Duration: 5.81953667s trafficmonitor_1 | 2020-06-21T21:21:02.475384200Z DEBUG: cache.go:183: 2020-06-21T21:21:02.475263732Z: poll 23 2020-06-21 21:21:02.475262662 +0000 UTC m=+59.949036293 start trafficmonitor_1 | 2020-06-21T21:21:02.476344037Z DEBUG: cache.go:195: 2020-06-21T21:21:02.476295637Z: poll 23 2020-06-21 21:21:02.47629522 +0000 UTC m=+59.950068846 poller end trafficmonitor_1 | 2020-06-21T21:21:02.476357457Z DEBUG: cache.go:279: 2020-06-21T21:21:02.476326457Z: poll 23 2020-06-21 21:21:02.476325898 +0000 UTC m=+59.950099522 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:02.476433149Z INFO: cache.go:49: 2020-06-21T21:21:02.476390832Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:02.476483183Z DEBUG: health.go:157: 2020-06-21T21:21:02.476450996Z: poll 23 2020-06-21 21:21:02.476450712 +0000 UTC m=+59.950224334 finish trafficmonitor_1 | 2020-06-21T21:21:03.135400513Z DEBUG: cache.go:177: 2020-06-21T21:21:03.135235796Z: Intended Duration: 2.91s Actual Duration: 5.819545209s trafficmonitor_1 | 2020-06-21T21:21:03.135423723Z DEBUG: cache.go:183: 2020-06-21T21:21:03.135268028Z: poll 24 2020-06-21 21:21:03.135266006 +0000 UTC m=+60.609039627 start trafficmonitor_1 | 2020-06-21T21:21:03.136568944Z DEBUG: cache.go:195: 2020-06-21T21:21:03.136472549Z: poll 24 2020-06-21 21:21:03.136472123 +0000 UTC m=+60.610245746 poller end trafficmonitor_1 | 2020-06-21T21:21:03.136594119Z DEBUG: cache.go:279: 2020-06-21T21:21:03.136502051Z: poll 24 2020-06-21 21:21:03.136494812 +0000 UTC m=+60.610268436 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:03.136668791Z INFO: cache.go:49: 2020-06-21T21:21:03.136628842Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:03.136718041Z DEBUG: health.go:157: 2020-06-21T21:21:03.136687499Z: poll 24 2020-06-21 21:21:03.136687067 +0000 UTC m=+60.610460700 finish trafficmonitor_1 | 2020-06-21T21:21:08.295625320Z DEBUG: cache.go:177: 2020-06-21T21:21:08.295327823Z: Intended Duration: 2.91s Actual Duration: 5.820059759s trafficmonitor_1 | 2020-06-21T21:21:08.295681698Z DEBUG: cache.go:183: 2020-06-21T21:21:08.295446237Z: poll 25 2020-06-21 21:21:08.295442047 +0000 UTC m=+65.769215753 start trafficmonitor_1 | 2020-06-21T21:21:08.299646240Z DEBUG: cache.go:195: 2020-06-21T21:21:08.299375899Z: poll 25 2020-06-21 21:21:08.29937385 +0000 UTC m=+65.773147576 poller end trafficmonitor_1 | 2020-06-21T21:21:08.299714477Z DEBUG: cache.go:279: 2020-06-21T21:21:08.299492736Z: poll 25 2020-06-21 21:21:08.299489406 +0000 UTC m=+65.773263194 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:08.300014718Z INFO: cache.go:49: 2020-06-21T21:21:08.29972906Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:08.300123379Z DEBUG: health.go:157: 2020-06-21T21:21:08.299992878Z: poll 25 2020-06-21 21:21:08.299991398 +0000 UTC m=+65.773765112 finish trafficmonitor_1 | 2020-06-21T21:21:08.955347134Z DEBUG: cache.go:177: 2020-06-21T21:21:08.955242416Z: Intended Duration: 2.91s Actual Duration: 5.819975222s trafficmonitor_1 | 2020-06-21T21:21:08.955532716Z DEBUG: cache.go:183: 2020-06-21T21:21:08.955269669Z: poll 26 2020-06-21 21:21:08.955269013 +0000 UTC m=+66.429042637 start trafficmonitor_1 | 2020-06-21T21:21:08.956506019Z DEBUG: cache.go:195: 2020-06-21T21:21:08.956397331Z: poll 26 2020-06-21 21:21:08.956396798 +0000 UTC m=+66.430170429 poller end trafficmonitor_1 | 2020-06-21T21:21:08.956515201Z DEBUG: cache.go:279: 2020-06-21T21:21:08.956436226Z: poll 26 2020-06-21 21:21:08.956435732 +0000 UTC m=+66.430209365 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:08.956561303Z INFO: cache.go:49: 2020-06-21T21:21:08.956517482Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:08.956617815Z DEBUG: health.go:157: 2020-06-21T21:21:08.956591673Z: poll 26 2020-06-21 21:21:08.956591404 +0000 UTC m=+66.430365027 finish trafficmonitor_1 | 2020-06-21T21:21:12.403923448Z DEBUG: cache.go:177: 2020-06-21T21:21:12.403290705Z: Intended Duration: 5.82s Actual Duration: 11.639916091s trafficmonitor_1 | 2020-06-21T21:21:12.403982434Z DEBUG: cache.go:183: 2020-06-21T21:21:12.40374232Z: poll 27 2020-06-21 21:21:12.40373936 +0000 UTC m=+69.877513081 start trafficmonitor_1 | 2020-06-21T21:21:12.408847346Z DEBUG: cache.go:195: 2020-06-21T21:21:12.40858334Z: poll 27 2020-06-21 21:21:12.408581426 +0000 UTC m=+69.882355156 poller end trafficmonitor_1 | 2020-06-21T21:21:12.408898337Z DEBUG: cache.go:279: 2020-06-21T21:21:12.408694911Z: poll 27 2020-06-21 21:21:12.408692474 +0000 UTC m=+69.882466199 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:12.411567823Z INFO: astats.go:134: 2020-06-21T21:21:12.411261056Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.411609890Z INFO: astats.go:134: 2020-06-21T21:21:12.411375603Z: precomputing cache mid stat system.proc.loadavg value 1.02 1.78 1.55 1/2211 403 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.411721483Z INFO: astats.go:134: 2020-06-21T21:21:12.411533457Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412268130Z INFO: astats.go:134: 2020-06-21T21:21:12.411978539Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412307845Z INFO: astats.go:134: 2020-06-21T21:21:12.412035215Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412358936Z INFO: astats.go:134: 2020-06-21T21:21:12.412078693Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412372471Z INFO: astats.go:134: 2020-06-21T21:21:12.412103431Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412384580Z INFO: astats.go:134: 2020-06-21T21:21:12.412131779Z: precomputing cache mid stat system.proc.net.dev value eth0: 655039 1356 0 0 0 0 0 0 359397 1552 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412412300Z INFO: astats.go:134: 2020-06-21T21:21:12.412226048Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412430117Z INFO: astats.go:134: 2020-06-21T21:21:12.412312709Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:12.412586113Z INFO: stat.go:165: 2020-06-21T21:21:12.41238009Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:12.412619615Z INFO: stat.go:181: 2020-06-21T21:21:12.412429734Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:12.412793414Z INFO: cache.go:49: 2020-06-21T21:21:12.412580331Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:12.413092280Z INFO: monitorconfig.go:263: 2020-06-21T21:21:12.41287362Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:12.413121371Z INFO: monitorconfig.go:269: 2020-06-21T21:21:12.412938662Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:12.413279989Z INFO: monitorconfig.go:263: 2020-06-21T21:21:12.413086139Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:12.413324954Z INFO: monitorconfig.go:269: 2020-06-21T21:21:12.41312343Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:12.414927342Z ERROR: stat.go:292: 2020-06-21T21:21:12.414680851Z: Adding result from mid: some stats could not be added: cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:12.414969291Z INFO: stat.go:54: 2020-06-21T21:21:12.414768115Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:12.414986708Z INFO: stat.go:458: 2020-06-21T21:21:12.414835352Z: CreateStats took 78.101µs trafficmonitor_1 | 2020-06-21T21:21:12.415227884Z INFO: stat.go:200: 2020-06-21T21:21:12.415041677Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:13.929265091Z DEBUG: cache.go:177: 2020-06-21T21:21:13.928794261Z: Intended Duration: 5.82s Actual Duration: 11.640061023s trafficmonitor_1 | 2020-06-21T21:21:13.929325702Z DEBUG: cache.go:183: 2020-06-21T21:21:13.929088902Z: poll 28 2020-06-21 21:21:13.929086082 +0000 UTC m=+71.402859788 start trafficmonitor_1 | 2020-06-21T21:21:13.933789437Z DEBUG: cache.go:195: 2020-06-21T21:21:13.933570683Z: poll 28 2020-06-21 21:21:13.933568774 +0000 UTC m=+71.407342483 poller end trafficmonitor_1 | 2020-06-21T21:21:13.933833958Z DEBUG: cache.go:279: 2020-06-21T21:21:13.933681897Z: poll 28 2020-06-21 21:21:13.933679406 +0000 UTC m=+71.407453128 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:13.936127633Z INFO: astats.go:134: 2020-06-21T21:21:13.935928123Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936163990Z INFO: astats.go:134: 2020-06-21T21:21:13.936011119Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936185180Z INFO: astats.go:134: 2020-06-21T21:21:13.936061456Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936416958Z INFO: astats.go:134: 2020-06-21T21:21:13.936254569Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936594771Z INFO: astats.go:134: 2020-06-21T21:21:13.93646944Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936713929Z INFO: astats.go:134: 2020-06-21T21:21:13.936587436Z: precomputing cache edge stat system.proc.net.dev value eth0: 276914 655 0 0 0 0 0 0 238315 698 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936748971Z INFO: astats.go:134: 2020-06-21T21:21:13.936641209Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936865479Z INFO: astats.go:134: 2020-06-21T21:21:13.936723501Z: precomputing cache edge stat system.proc.loadavg value 1.02 1.78 1.55 2/2211 399 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936943193Z INFO: astats.go:134: 2020-06-21T21:21:13.936808556Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.936986852Z INFO: astats.go:134: 2020-06-21T21:21:13.936879839Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:13.937060999Z INFO: stat.go:165: 2020-06-21T21:21:13.936960504Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:13.937081753Z INFO: stat.go:181: 2020-06-21T21:21:13.936989953Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:13.937163135Z INFO: cache.go:49: 2020-06-21T21:21:13.937060975Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:13.937545152Z INFO: monitorconfig.go:263: 2020-06-21T21:21:13.937416303Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:13.937565093Z INFO: monitorconfig.go:269: 2020-06-21T21:21:13.93747622Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:13.937767629Z INFO: monitorconfig.go:263: 2020-06-21T21:21:13.937609664Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:13.937800477Z INFO: monitorconfig.go:269: 2020-06-21T21:21:13.937648994Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:13.938941354Z ERROR: stat.go:292: 2020-06-21T21:21:13.938799959Z: Adding result from edge: some stats could not be added: cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:13.939063225Z INFO: stat.go:54: 2020-06-21T21:21:13.93886935Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:13.939083424Z INFO: stat.go:458: 2020-06-21T21:21:13.938923243Z: CreateStats took 61.243µs trafficmonitor_1 | 2020-06-21T21:21:13.939295577Z INFO: stat.go:200: 2020-06-21T21:21:13.939069807Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:14.115577789Z DEBUG: cache.go:177: 2020-06-21T21:21:14.115294188Z: Intended Duration: 2.91s Actual Duration: 5.819851187s trafficmonitor_1 | 2020-06-21T21:21:14.115607536Z DEBUG: cache.go:183: 2020-06-21T21:21:14.115491362Z: poll 29 2020-06-21 21:21:14.115489546 +0000 UTC m=+71.589263198 start trafficmonitor_1 | 2020-06-21T21:21:14.117582770Z DEBUG: cache.go:195: 2020-06-21T21:21:14.117431668Z: poll 29 2020-06-21 21:21:14.117430751 +0000 UTC m=+71.591204408 poller end trafficmonitor_1 | 2020-06-21T21:21:14.117607290Z DEBUG: cache.go:279: 2020-06-21T21:21:14.117489705Z: poll 29 2020-06-21 21:21:14.117488513 +0000 UTC m=+71.591262166 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:14.117775331Z INFO: cache.go:49: 2020-06-21T21:21:14.117604439Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:14.117864566Z DEBUG: health.go:157: 2020-06-21T21:21:14.117745864Z: poll 29 2020-06-21 21:21:14.117744877 +0000 UTC m=+71.591518566 finish trafficmonitor_1 | 2020-06-21T21:21:14.775721396Z DEBUG: cache.go:177: 2020-06-21T21:21:14.775353107Z: Intended Duration: 2.91s Actual Duration: 5.820077389s trafficmonitor_1 | 2020-06-21T21:21:14.776062678Z DEBUG: cache.go:183: 2020-06-21T21:21:14.775699461Z: poll 30 2020-06-21 21:21:14.775694909 +0000 UTC m=+72.249468629 start trafficmonitor_1 | 2020-06-21T21:21:14.780020314Z DEBUG: cache.go:195: 2020-06-21T21:21:14.779743692Z: poll 30 2020-06-21 21:21:14.779741017 +0000 UTC m=+72.253514744 poller end trafficmonitor_1 | 2020-06-21T21:21:14.780061529Z DEBUG: cache.go:279: 2020-06-21T21:21:14.779871264Z: poll 30 2020-06-21 21:21:14.779868461 +0000 UTC m=+72.253642182 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:14.780490909Z INFO: cache.go:49: 2020-06-21T21:21:14.780154562Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:14.780606359Z DEBUG: health.go:157: 2020-06-21T21:21:14.780416404Z: poll 30 2020-06-21 21:21:14.78041488 +0000 UTC m=+72.254188601 finish trafficmonitor_1 | 2020-06-21T21:21:19.935643750Z DEBUG: cache.go:177: 2020-06-21T21:21:19.935363201Z: Intended Duration: 2.91s Actual Duration: 5.819870557s trafficmonitor_1 | 2020-06-21T21:21:19.935724069Z DEBUG: cache.go:183: 2020-06-21T21:21:19.935447791Z: poll 31 2020-06-21 21:21:19.935445666 +0000 UTC m=+77.409219369 start trafficmonitor_1 | 2020-06-21T21:21:19.939466106Z DEBUG: cache.go:195: 2020-06-21T21:21:19.939205487Z: poll 31 2020-06-21 21:21:19.939202983 +0000 UTC m=+77.412976694 poller end trafficmonitor_1 | 2020-06-21T21:21:19.939502444Z DEBUG: cache.go:279: 2020-06-21T21:21:19.939317823Z: poll 31 2020-06-21 21:21:19.939315131 +0000 UTC m=+77.413088849 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:19.939872882Z INFO: cache.go:49: 2020-06-21T21:21:19.939546537Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:19.940005989Z DEBUG: health.go:157: 2020-06-21T21:21:19.939828942Z: poll 31 2020-06-21 21:21:19.939827658 +0000 UTC m=+77.413601373 finish trafficmonitor_1 | 2020-06-21T21:21:20.596033225Z DEBUG: cache.go:177: 2020-06-21T21:21:20.595358965Z: Intended Duration: 2.91s Actual Duration: 5.819658986s trafficmonitor_1 | 2020-06-21T21:21:20.596093437Z DEBUG: cache.go:183: 2020-06-21T21:21:20.595859366Z: poll 32 2020-06-21 21:21:20.595851914 +0000 UTC m=+78.069625667 start trafficmonitor_1 | 2020-06-21T21:21:20.599942068Z DEBUG: cache.go:195: 2020-06-21T21:21:20.599693257Z: poll 32 2020-06-21 21:21:20.599691005 +0000 UTC m=+78.073464736 poller end trafficmonitor_1 | 2020-06-21T21:21:20.599989655Z DEBUG: cache.go:279: 2020-06-21T21:21:20.599832461Z: poll 32 2020-06-21 21:21:20.599829391 +0000 UTC m=+78.073603100 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:20.600183685Z INFO: cache.go:49: 2020-06-21T21:21:20.600046064Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:20.600409574Z DEBUG: health.go:157: 2020-06-21T21:21:20.600259263Z: poll 32 2020-06-21 21:21:20.60025814 +0000 UTC m=+78.074031844 finish trafficmonitor_1 | 2020-06-21T21:21:24.043454738Z DEBUG: cache.go:177: 2020-06-21T21:21:24.043199687Z: Intended Duration: 5.82s Actual Duration: 11.639458647s trafficmonitor_1 | 2020-06-21T21:21:24.043509303Z DEBUG: cache.go:183: 2020-06-21T21:21:24.043285528Z: poll 33 2020-06-21 21:21:24.043283232 +0000 UTC m=+81.517056930 start trafficmonitor_1 | 2020-06-21T21:21:24.048299489Z DEBUG: cache.go:195: 2020-06-21T21:21:24.04804431Z: poll 33 2020-06-21 21:21:24.048042488 +0000 UTC m=+81.521816231 poller end trafficmonitor_1 | 2020-06-21T21:21:24.048340174Z DEBUG: cache.go:279: 2020-06-21T21:21:24.048152081Z: poll 33 2020-06-21 21:21:24.048149804 +0000 UTC m=+81.521923514 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:24.050760263Z INFO: astats.go:134: 2020-06-21T21:21:24.050516484Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051040333Z INFO: astats.go:134: 2020-06-21T21:21:24.050821566Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051068275Z INFO: astats.go:134: 2020-06-21T21:21:24.050893862Z: precomputing cache mid stat system.proc.net.dev value eth0: 656128 1365 0 0 0 0 0 0 397474 1558 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051086548Z INFO: astats.go:134: 2020-06-21T21:21:24.050946738Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051279549Z INFO: astats.go:134: 2020-06-21T21:21:24.051047962Z: precomputing cache mid stat system.proc.loadavg value 0.86 1.72 1.53 2/2212 405 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051313607Z INFO: astats.go:134: 2020-06-21T21:21:24.051077497Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051380128Z INFO: astats.go:134: 2020-06-21T21:21:24.051195977Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051418686Z INFO: astats.go:134: 2020-06-21T21:21:24.051257422Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051459545Z INFO: astats.go:134: 2020-06-21T21:21:24.05132489Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051484313Z INFO: astats.go:134: 2020-06-21T21:21:24.051385138Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:24.051691595Z INFO: stat.go:165: 2020-06-21T21:21:24.051466335Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:24.051724795Z INFO: stat.go:181: 2020-06-21T21:21:24.05149509Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:24.051797147Z INFO: cache.go:49: 2020-06-21T21:21:24.051614058Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:24.052130613Z INFO: monitorconfig.go:263: 2020-06-21T21:21:24.051908611Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:24.052163411Z INFO: monitorconfig.go:269: 2020-06-21T21:21:24.051970233Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:24.052334281Z INFO: monitorconfig.go:263: 2020-06-21T21:21:24.052107121Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:24.052366809Z INFO: monitorconfig.go:269: 2020-06-21T21:21:24.052142136Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:24.053547437Z ERROR: stat.go:292: 2020-06-21T21:21:24.053359901Z: Adding result from mid: some stats could not be added: cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:24.053579625Z INFO: stat.go:54: 2020-06-21T21:21:24.053463042Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:24.053655137Z INFO: stat.go:458: 2020-06-21T21:21:24.053532482Z: CreateStats took 110.561µs trafficmonitor_1 | 2020-06-21T21:21:24.053787818Z INFO: stat.go:200: 2020-06-21T21:21:24.053689095Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:25.569355768Z DEBUG: cache.go:177: 2020-06-21T21:21:25.568767398Z: Intended Duration: 5.82s Actual Duration: 11.639680213s trafficmonitor_1 | 2020-06-21T21:21:25.569441530Z DEBUG: cache.go:183: 2020-06-21T21:21:25.569149297Z: poll 34 2020-06-21 21:21:25.56914656 +0000 UTC m=+83.042920272 start trafficmonitor_1 | 2020-06-21T21:21:25.574237409Z DEBUG: cache.go:195: 2020-06-21T21:21:25.574034243Z: poll 34 2020-06-21 21:21:25.574032021 +0000 UTC m=+83.047805738 poller end trafficmonitor_1 | 2020-06-21T21:21:25.574347060Z DEBUG: cache.go:279: 2020-06-21T21:21:25.574141707Z: poll 34 2020-06-21 21:21:25.574139446 +0000 UTC m=+83.047913163 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:25.577077886Z INFO: astats.go:134: 2020-06-21T21:21:25.57673694Z: precomputing cache edge stat system.proc.net.dev value eth0: 278006 664 0 0 0 0 0 0 276397 704 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577123791Z INFO: astats.go:134: 2020-06-21T21:21:25.576967222Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577417850Z INFO: astats.go:134: 2020-06-21T21:21:25.577194331Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577457180Z INFO: astats.go:134: 2020-06-21T21:21:25.577310082Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577702552Z INFO: astats.go:134: 2020-06-21T21:21:25.577507938Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577736533Z INFO: astats.go:134: 2020-06-21T21:21:25.577609096Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577880331Z INFO: astats.go:134: 2020-06-21T21:21:25.577675564Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.577904771Z INFO: astats.go:134: 2020-06-21T21:21:25.577809349Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.578019804Z INFO: astats.go:134: 2020-06-21T21:21:25.577861241Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.578053142Z INFO: astats.go:134: 2020-06-21T21:21:25.577918815Z: precomputing cache edge stat system.proc.loadavg value 1.04 1.75 1.54 1/2213 400 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:25.578163217Z INFO: stat.go:165: 2020-06-21T21:21:25.57802135Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:25.578185437Z INFO: stat.go:181: 2020-06-21T21:21:25.578076767Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:25.578322251Z INFO: cache.go:49: 2020-06-21T21:21:25.578183291Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:25.578763748Z INFO: monitorconfig.go:263: 2020-06-21T21:21:25.578571315Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:25.578828271Z INFO: monitorconfig.go:269: 2020-06-21T21:21:25.578656048Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:25.578965260Z INFO: monitorconfig.go:263: 2020-06-21T21:21:25.578803121Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:25.578986150Z INFO: monitorconfig.go:269: 2020-06-21T21:21:25.578841915Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:25.579359435Z ERROR: stat.go:292: 2020-06-21T21:21:25.579272376Z: Adding result from edge: some stats could not be added: cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:25.579364758Z INFO: stat.go:54: 2020-06-21T21:21:25.579290449Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:25.579366967Z INFO: stat.go:458: 2020-06-21T21:21:25.57930208Z: CreateStats took 13.792µs trafficmonitor_1 | 2020-06-21T21:21:25.579380227Z INFO: stat.go:200: 2020-06-21T21:21:25.579337139Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:25.755658680Z DEBUG: cache.go:177: 2020-06-21T21:21:25.75536838Z: Intended Duration: 2.91s Actual Duration: 5.819920017s trafficmonitor_1 | 2020-06-21T21:21:25.755705319Z DEBUG: cache.go:183: 2020-06-21T21:21:25.755459412Z: poll 35 2020-06-21 21:21:25.755457192 +0000 UTC m=+83.229230911 start trafficmonitor_1 | 2020-06-21T21:21:25.759245122Z DEBUG: cache.go:195: 2020-06-21T21:21:25.758981549Z: poll 35 2020-06-21 21:21:25.758976935 +0000 UTC m=+83.232750649 poller end trafficmonitor_1 | 2020-06-21T21:21:25.759284614Z DEBUG: cache.go:279: 2020-06-21T21:21:25.759107972Z: poll 35 2020-06-21 21:21:25.759105229 +0000 UTC m=+83.232878942 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:25.759555156Z INFO: cache.go:49: 2020-06-21T21:21:25.759338751Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:25.759769681Z DEBUG: health.go:157: 2020-06-21T21:21:25.759581419Z: poll 35 2020-06-21 21:21:25.759580291 +0000 UTC m=+83.233353992 finish trafficmonitor_1 | 2020-06-21T21:21:26.415680143Z DEBUG: cache.go:177: 2020-06-21T21:21:26.41533915Z: Intended Duration: 2.91s Actual Duration: 5.819481012s trafficmonitor_1 | 2020-06-21T21:21:26.415728760Z DEBUG: cache.go:183: 2020-06-21T21:21:26.41544985Z: poll 36 2020-06-21 21:21:26.415447675 +0000 UTC m=+83.889221381 start trafficmonitor_1 | 2020-06-21T21:21:26.419347582Z DEBUG: cache.go:195: 2020-06-21T21:21:26.41905852Z: poll 36 2020-06-21 21:21:26.419056341 +0000 UTC m=+83.892830049 poller end trafficmonitor_1 | 2020-06-21T21:21:26.419400785Z DEBUG: cache.go:279: 2020-06-21T21:21:26.41918003Z: poll 36 2020-06-21 21:21:26.419177679 +0000 UTC m=+83.892951392 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:26.419691926Z INFO: cache.go:49: 2020-06-21T21:21:26.419377661Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:26.419780820Z DEBUG: health.go:157: 2020-06-21T21:21:26.41963656Z: poll 36 2020-06-21 21:21:26.419635296 +0000 UTC m=+83.893408999 finish trafficmonitor_1 | 2020-06-21T21:21:31.575720690Z DEBUG: cache.go:177: 2020-06-21T21:21:31.575378134Z: Intended Duration: 2.91s Actual Duration: 5.819918177s trafficmonitor_1 | 2020-06-21T21:21:31.575813720Z DEBUG: cache.go:183: 2020-06-21T21:21:31.575476837Z: poll 37 2020-06-21 21:21:31.575474075 +0000 UTC m=+89.049247793 start trafficmonitor_1 | 2020-06-21T21:21:31.579959353Z DEBUG: cache.go:195: 2020-06-21T21:21:31.579678893Z: poll 37 2020-06-21 21:21:31.579676751 +0000 UTC m=+89.053450468 poller end trafficmonitor_1 | 2020-06-21T21:21:31.580009219Z DEBUG: cache.go:279: 2020-06-21T21:21:31.579809577Z: poll 37 2020-06-21 21:21:31.579806312 +0000 UTC m=+89.053580063 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:31.580407456Z INFO: cache.go:49: 2020-06-21T21:21:31.580065113Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:31.580532626Z DEBUG: health.go:157: 2020-06-21T21:21:31.580357699Z: poll 37 2020-06-21 21:21:31.58035637 +0000 UTC m=+89.054130088 finish trafficmonitor_1 | 2020-06-21T21:21:32.235695418Z DEBUG: cache.go:177: 2020-06-21T21:21:32.235354003Z: Intended Duration: 2.91s Actual Duration: 5.819903703s trafficmonitor_1 | 2020-06-21T21:21:32.235768003Z DEBUG: cache.go:183: 2020-06-21T21:21:32.235449717Z: poll 38 2020-06-21 21:21:32.235447095 +0000 UTC m=+89.709220805 start trafficmonitor_1 | 2020-06-21T21:21:32.240083965Z DEBUG: cache.go:195: 2020-06-21T21:21:32.239778418Z: poll 38 2020-06-21 21:21:32.239776141 +0000 UTC m=+89.713549896 poller end trafficmonitor_1 | 2020-06-21T21:21:32.240155873Z DEBUG: cache.go:279: 2020-06-21T21:21:32.239932785Z: poll 38 2020-06-21 21:21:32.239929527 +0000 UTC m=+89.713703246 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:32.240441749Z INFO: cache.go:49: 2020-06-21T21:21:32.24020264Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:32.240578705Z DEBUG: health.go:157: 2020-06-21T21:21:32.240380928Z: poll 38 2020-06-21 21:21:32.240379528 +0000 UTC m=+89.714153244 finish trafficmonitor_1 | 2020-06-21T21:21:35.683995327Z DEBUG: cache.go:177: 2020-06-21T21:21:35.683279372Z: Intended Duration: 5.82s Actual Duration: 11.639989613s trafficmonitor_1 | 2020-06-21T21:21:35.684054758Z DEBUG: cache.go:183: 2020-06-21T21:21:35.683760931Z: poll 39 2020-06-21 21:21:35.683757781 +0000 UTC m=+93.157531492 start trafficmonitor_1 | 2020-06-21T21:21:35.689005942Z DEBUG: cache.go:195: 2020-06-21T21:21:35.688702685Z: poll 39 2020-06-21 21:21:35.688700526 +0000 UTC m=+93.162474236 poller end trafficmonitor_1 | 2020-06-21T21:21:35.689048207Z DEBUG: cache.go:279: 2020-06-21T21:21:35.688828749Z: poll 39 2020-06-21 21:21:35.68882601 +0000 UTC m=+93.162599727 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:35.692326001Z INFO: astats.go:134: 2020-06-21T21:21:35.692025027Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.692368864Z INFO: astats.go:134: 2020-06-21T21:21:35.692102389Z: precomputing cache mid stat system.proc.loadavg value 0.88 1.69 1.52 2/2214 406 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.692382468Z INFO: astats.go:134: 2020-06-21T21:21:35.692164793Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.692494292Z INFO: astats.go:134: 2020-06-21T21:21:35.692283331Z: precomputing cache mid stat system.proc.net.dev value eth0: 657217 1374 0 0 0 0 0 0 435554 1564 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.692555290Z INFO: astats.go:134: 2020-06-21T21:21:35.692377414Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.692750555Z INFO: astats.go:134: 2020-06-21T21:21:35.692520712Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.693044973Z INFO: astats.go:134: 2020-06-21T21:21:35.692805521Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.693080549Z INFO: astats.go:134: 2020-06-21T21:21:35.692883247Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.693149889Z INFO: astats.go:134: 2020-06-21T21:21:35.692949519Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.693186415Z INFO: astats.go:134: 2020-06-21T21:21:35.693016785Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:35.693532317Z INFO: stat.go:165: 2020-06-21T21:21:35.693259021Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:35.693571177Z INFO: stat.go:181: 2020-06-21T21:21:35.693300524Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:35.693680370Z INFO: cache.go:49: 2020-06-21T21:21:35.693477647Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:35.694030138Z INFO: monitorconfig.go:263: 2020-06-21T21:21:35.69381261Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:35.694054484Z INFO: monitorconfig.go:269: 2020-06-21T21:21:35.693878804Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:35.694227901Z INFO: monitorconfig.go:263: 2020-06-21T21:21:35.69402907Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:35.694249636Z INFO: monitorconfig.go:269: 2020-06-21T21:21:35.694081541Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:35.695695929Z ERROR: stat.go:292: 2020-06-21T21:21:35.695491971Z: Adding result from mid: some stats could not be added: cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:35.695721827Z INFO: stat.go:54: 2020-06-21T21:21:35.695568868Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:35.695732683Z INFO: stat.go:458: 2020-06-21T21:21:35.695620079Z: CreateStats took 60.185µs trafficmonitor_1 | 2020-06-21T21:21:35.695964399Z INFO: stat.go:200: 2020-06-21T21:21:35.695781068Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:37.209321877Z DEBUG: cache.go:177: 2020-06-21T21:21:37.208796498Z: Intended Duration: 5.82s Actual Duration: 11.639646879s trafficmonitor_1 | 2020-06-21T21:21:37.209376210Z DEBUG: cache.go:183: 2020-06-21T21:21:37.209131485Z: poll 40 2020-06-21 21:21:37.20912863 +0000 UTC m=+94.682902340 start trafficmonitor_1 | 2020-06-21T21:21:37.215006966Z DEBUG: cache.go:195: 2020-06-21T21:21:37.214786546Z: poll 40 2020-06-21 21:21:37.214784504 +0000 UTC m=+94.688558239 poller end trafficmonitor_1 | 2020-06-21T21:21:37.215079816Z DEBUG: cache.go:279: 2020-06-21T21:21:37.214911005Z: poll 40 2020-06-21 21:21:37.214906573 +0000 UTC m=+94.688680357 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:37.217712653Z INFO: astats.go:134: 2020-06-21T21:21:37.217477348Z: precomputing cache edge stat system.proc.loadavg value 0.88 1.69 1.52 1/2213 401 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.217755240Z INFO: astats.go:134: 2020-06-21T21:21:37.217573117Z: precomputing cache edge stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.217818362Z INFO: astats.go:134: 2020-06-21T21:21:37.21766068Z: precomputing cache edge stat system.astatsLoad value 1592774402 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.217981212Z INFO: astats.go:134: 2020-06-21T21:21:37.217792366Z: precomputing cache edge stat system.proc.net.dev value eth0: 279098 673 0 0 0 0 0 0 314484 710 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.218087135Z INFO: astats.go:134: 2020-06-21T21:21:37.21792375Z: precomputing cache edge stat system.lastReload value 1592774402 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.218315453Z INFO: astats.go:134: 2020-06-21T21:21:37.218144634Z: precomputing cache edge stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.218608391Z INFO: astats.go:134: 2020-06-21T21:21:37.218409959Z: precomputing cache edge stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.218714872Z INFO: astats.go:134: 2020-06-21T21:21:37.218531454Z: precomputing cache edge stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.218874231Z INFO: astats.go:134: 2020-06-21T21:21:37.218739361Z: precomputing cache edge stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.218970052Z INFO: astats.go:134: 2020-06-21T21:21:37.218842659Z: precomputing cache edge stat system.lastReloadRequest value 1592774402 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:37.219195279Z INFO: stat.go:165: 2020-06-21T21:21:37.218971413Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:37.219232603Z INFO: stat.go:181: 2020-06-21T21:21:37.219033846Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:37.219255011Z INFO: cache.go:49: 2020-06-21T21:21:37.219132845Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:37.219623755Z INFO: monitorconfig.go:263: 2020-06-21T21:21:37.219484367Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:37.219677979Z INFO: monitorconfig.go:269: 2020-06-21T21:21:37.219547935Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:37.219839181Z INFO: monitorconfig.go:263: 2020-06-21T21:21:37.219721592Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:37.219859468Z INFO: monitorconfig.go:269: 2020-06-21T21:21:37.21976491Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:37.221426690Z ERROR: stat.go:292: 2020-06-21T21:21:37.221202673Z: Adding result from edge: some stats could not be added: cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:37.221469717Z INFO: stat.go:54: 2020-06-21T21:21:37.221298057Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:37.221602333Z INFO: stat.go:458: 2020-06-21T21:21:37.221391253Z: CreateStats took 105.675µs trafficmonitor_1 | 2020-06-21T21:21:37.221758015Z INFO: stat.go:200: 2020-06-21T21:21:37.221610079Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer trafficmonitor_1 | 2020-06-21T21:21:37.395799563Z DEBUG: cache.go:177: 2020-06-21T21:21:37.39528846Z: Intended Duration: 2.91s Actual Duration: 5.819808802s trafficmonitor_1 | 2020-06-21T21:21:37.395861775Z DEBUG: cache.go:183: 2020-06-21T21:21:37.395608108Z: poll 41 2020-06-21 21:21:37.39560497 +0000 UTC m=+94.869378691 start trafficmonitor_1 | 2020-06-21T21:21:37.399722686Z DEBUG: cache.go:195: 2020-06-21T21:21:37.399392714Z: poll 41 2020-06-21 21:21:37.39939081 +0000 UTC m=+94.873164521 poller end trafficmonitor_1 | 2020-06-21T21:21:37.399764807Z DEBUG: cache.go:279: 2020-06-21T21:21:37.399517776Z: poll 41 2020-06-21 21:21:37.399515273 +0000 UTC m=+94.873289004 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:37.400025093Z INFO: cache.go:49: 2020-06-21T21:21:37.399726625Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:37.400137609Z DEBUG: health.go:157: 2020-06-21T21:21:37.399965557Z: poll 41 2020-06-21 21:21:37.399964387 +0000 UTC m=+94.873738096 finish trafficmonitor_1 | 2020-06-21T21:21:38.055634502Z DEBUG: cache.go:177: 2020-06-21T21:21:38.055360175Z: Intended Duration: 2.91s Actual Duration: 5.819910227s trafficmonitor_1 | 2020-06-21T21:21:38.055992480Z DEBUG: cache.go:183: 2020-06-21T21:21:38.055765639Z: poll 42 2020-06-21 21:21:38.055752106 +0000 UTC m=+95.529525828 start trafficmonitor_1 | 2020-06-21T21:21:38.059861165Z DEBUG: cache.go:195: 2020-06-21T21:21:38.059669096Z: poll 42 2020-06-21 21:21:38.059665983 +0000 UTC m=+95.533439700 poller end trafficmonitor_1 | 2020-06-21T21:21:38.059980540Z DEBUG: cache.go:279: 2020-06-21T21:21:38.059790204Z: poll 42 2020-06-21 21:21:38.05978799 +0000 UTC m=+95.533561715 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:38.060320680Z INFO: cache.go:49: 2020-06-21T21:21:38.060076691Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:38.060592114Z DEBUG: health.go:157: 2020-06-21T21:21:38.060344859Z: poll 42 2020-06-21 21:21:38.06034293 +0000 UTC m=+95.534116683 finish trafficmonitor_1 | 2020-06-21T21:21:43.215614918Z DEBUG: cache.go:177: 2020-06-21T21:21:43.215316665Z: Intended Duration: 2.91s Actual Duration: 5.819707487s trafficmonitor_1 | 2020-06-21T21:21:43.215676342Z DEBUG: cache.go:183: 2020-06-21T21:21:43.215405097Z: poll 43 2020-06-21 21:21:43.215401929 +0000 UTC m=+100.689175630 start trafficmonitor_1 | 2020-06-21T21:21:43.219320953Z DEBUG: cache.go:195: 2020-06-21T21:21:43.219090164Z: poll 43 2020-06-21 21:21:43.219088461 +0000 UTC m=+100.692862158 poller end trafficmonitor_1 | 2020-06-21T21:21:43.219359985Z DEBUG: cache.go:279: 2020-06-21T21:21:43.219196828Z: poll 43 2020-06-21 21:21:43.219194535 +0000 UTC m=+100.692968240 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:43.219666316Z INFO: cache.go:49: 2020-06-21T21:21:43.219385903Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:43.219753330Z DEBUG: health.go:157: 2020-06-21T21:21:43.219585162Z: poll 43 2020-06-21 21:21:43.219584071 +0000 UTC m=+100.693357776 finish trafficmonitor_1 | 2020-06-21T21:21:43.875633472Z DEBUG: cache.go:177: 2020-06-21T21:21:43.875343342Z: Intended Duration: 2.91s Actual Duration: 5.819586747s trafficmonitor_1 | 2020-06-21T21:21:43.875695435Z DEBUG: cache.go:183: 2020-06-21T21:21:43.875435228Z: poll 44 2020-06-21 21:21:43.875432281 +0000 UTC m=+101.349205984 start trafficmonitor_1 | 2020-06-21T21:21:43.879526538Z DEBUG: cache.go:195: 2020-06-21T21:21:43.879310825Z: poll 44 2020-06-21 21:21:43.879308546 +0000 UTC m=+101.353082256 poller end trafficmonitor_1 | 2020-06-21T21:21:43.879659307Z DEBUG: cache.go:279: 2020-06-21T21:21:43.879419406Z: poll 44 2020-06-21 21:21:43.879416831 +0000 UTC m=+101.353190539 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:43.879789229Z INFO: cache.go:49: 2020-06-21T21:21:43.879639965Z: Cache 'edge' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:43.879931365Z DEBUG: health.go:157: 2020-06-21T21:21:43.879805072Z: poll 44 2020-06-21 21:21:43.879803919 +0000 UTC m=+101.353577624 finish trafficmonitor_1 | 2020-06-21T21:21:47.323393099Z DEBUG: cache.go:177: 2020-06-21T21:21:47.323177362Z: Intended Duration: 5.82s Actual Duration: 11.639417947s trafficmonitor_1 | 2020-06-21T21:21:47.323420130Z DEBUG: cache.go:183: 2020-06-21T21:21:47.323227435Z: poll 45 2020-06-21 21:21:47.323226063 +0000 UTC m=+104.796999715 start trafficmonitor_1 | 2020-06-21T21:21:47.325393985Z DEBUG: cache.go:195: 2020-06-21T21:21:47.325219453Z: poll 45 2020-06-21 21:21:47.325217121 +0000 UTC m=+104.798990769 poller end trafficmonitor_1 | 2020-06-21T21:21:47.325411318Z DEBUG: cache.go:279: 2020-06-21T21:21:47.325279924Z: poll 45 2020-06-21 21:21:47.325278131 +0000 UTC m=+104.799051799 (format 'astats') handle start trafficmonitor_1 | 2020-06-21T21:21:47.326473564Z INFO: astats.go:134: 2020-06-21T21:21:47.326334138Z: precomputing cache mid stat system.notAvailable value false error stat 'system.notAvailable' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326527825Z INFO: astats.go:134: 2020-06-21T21:21:47.32643379Z: precomputing cache mid stat system.proc.net.dev value eth0: 658306 1383 0 0 0 0 0 0 473637 1570 0 0 0 0 0 0 error stat 'system.proc.net.dev' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326544616Z INFO: astats.go:134: 2020-06-21T21:21:47.32648563Z: precomputing cache mid stat system.astatsLoad value 1592774406 error stat 'system.astatsLoad' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326619212Z INFO: astats.go:134: 2020-06-21T21:21:47.326524011Z: precomputing cache mid stat system.proc.loadavg value 0.97 1.68 1.52 1/2214 407 error stat 'system.proc.loadavg' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326634151Z INFO: astats.go:134: 2020-06-21T21:21:47.32654942Z: precomputing cache mid stat system.inf.speed value 10000 error stat 'system.inf.speed' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326664335Z INFO: astats.go:134: 2020-06-21T21:21:47.326584747Z: precomputing cache mid stat system.configReloads value 1 error stat 'system.configReloads' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326675572Z INFO: astats.go:134: 2020-06-21T21:21:47.326601468Z: precomputing cache mid stat system.inf.name value eth0 error stat 'system.inf.name' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326706683Z INFO: astats.go:134: 2020-06-21T21:21:47.326661246Z: precomputing cache mid stat system.configReloadRequests value 1 error stat 'system.configReloadRequests' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326744387Z INFO: astats.go:134: 2020-06-21T21:21:47.326708475Z: precomputing cache mid stat system.lastReloadRequest value 1592774406 error stat 'system.lastReloadRequest' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326819357Z INFO: astats.go:134: 2020-06-21T21:21:47.32675176Z: precomputing cache mid stat system.lastReload value 1592774406 error stat 'system.lastReload' has unknown initial part 'system' trafficmonitor_1 | 2020-06-21T21:21:47.326860885Z INFO: stat.go:165: 2020-06-21T21:21:47.326814081Z: StatHistoryManager starting flushLoop with 1 results trafficmonitor_1 | 2020-06-21T21:21:47.326882999Z INFO: stat.go:181: 2020-06-21T21:21:47.326832348Z: StatHistoryManager flushLoop: stat chan is empty, processing 1 results trafficmonitor_1 | 2020-06-21T21:21:47.326936292Z INFO: cache.go:49: 2020-06-21T21:21:47.326896701Z: Cache 'mid' has 1 interfaces, we will use average (overflow possible) for vitality trafficmonitor_1 | 2020-06-21T21:21:47.327120954Z INFO: monitorconfig.go:263: 2020-06-21T21:21:47.327060665Z: health.polling.format for 'edge' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:47.327129655Z INFO: monitorconfig.go:269: 2020-06-21T21:21:47.327092859Z: health.polling.type for 'edge' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:47.327194155Z INFO: monitorconfig.go:263: 2020-06-21T21:21:47.327156226Z: health.polling.format for 'mid' is empty, using default 'astats' trafficmonitor_1 | 2020-06-21T21:21:47.327200996Z INFO: monitorconfig.go:269: 2020-06-21T21:21:47.327172292Z: health.polling.type for 'mid' is empty, using default 'http' trafficmonitor_1 | 2020-06-21T21:21:47.327851392Z ERROR: stat.go:292: 2020-06-21T21:21:47.327788678Z: Adding result from mid: some stats could not be added: cannot add stat system.inf.speed: incomparable stat type int; cannot add stat system.configReloads: incomparable stat type int; cannot add stat system.configReloadRequests: incomparable stat type int; cannot add stat system.lastReloadRequest: incomparable stat type int; cannot add stat system.lastReload: incomparable stat type int; cannot add stat system.astatsLoad: incomparable stat type int trafficmonitor_1 | 2020-06-21T21:21:47.327871263Z INFO: stat.go:54: 2020-06-21T21:21:47.327827602Z: CreateStats not adding availability data for 'mid': not found in DeliveryServices trafficmonitor_1 | 2020-06-21T21:21:47.327877960Z INFO: stat.go:458: 2020-06-21T21:21:47.327849813Z: CreateStats took 29.119µs trafficmonitor_1 | 2020-06-21T21:21:47.328012434Z INFO: stat.go:200: 2020-06-21T21:21:47.327943147Z: StatHistoryManager bufferLoop exiting flush loop, resetting buffer timer ``` </details> If the issue occurs when `TM_DEBUG_ENABLE` is set to `true` in `variables.env`, `traffic_monitor` will not immediately exit and will hang after ```go trafficmonitor_1 | 2020-06-22T14:49:51.858901616Z 2020-06-22T14:49:51Z error layer=rpc writing response:write tcp 127.0.0.1:2344->127.0.0.1:35216: use of closed network connection ``` If this happens, it will panic as soon as you try to connect your IDE to the `dlv` debugging process. TM can still be debugged by building the `trafficmonitor` CDN-in-a-Box image [with `'--continue'` omitted from the `dlv` command](https://github.com/apache/trafficcontrol/blob/e1d431b134/infrastructure/cdn-in-a-box/traffic_monitor/run.sh#L129). <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
