See <https://ci.trafficserver.apache.org/job/tsqa-master/285/changes>
Changes: [Bryan Call] Tool I use to analyze slow logs ------------------------------------------ [...truncated 13207 lines...] 10 686 3433.3 0 0 54505705/5450570 2206 2206 2206 91285565 12.9 0 10 695 3475.8 0 0 55279976/5527997 2239 2239 2239 91870914 13.9 0 10 704 3525.1 0 0 55286694/5528669 2270 2270 2270 91681540 14.9 0 10 715 3578.2 0 0 55220490/5522049 2302 2302 2302 91959069 15.9 0 10 720 3604.6 0 0 55550526/5555052 2321 2321 2322 92528308 16.9 0 10 726 3631.7 0 0 55116943/5511694 2343 2343 2342 92288893 17.9 0 10 724 3624.5 0 0 55277869/5527786 2342 2342 2342 92901058 18.9 0 10 729 3650.4 0 0 54850881/5485088 2354 2354 2354 91607213 19.9 0 con new ops 1B lat bytes/per svrs new ops total time err 10 735 3681.1 0 0 54784953/5478495 2372 2372 2372 91054047 20.9 0 10 744 3721.3 0 0 56055288/5605528 2387 2387 2386 92214019 21.9 0 10 750 3750.7 0 0 56259413/5625941 2404 2404 2404 93181153 22.9 0 10 747 3739.5 0 0 56769397/5676939 2398 2398 2399 93959266 23.9 0 10 746 3734.2 0 0 56562287/5656228 2390 2390 2390 93993785 24.9 0 10 746 3733.6 0 0 56101749/5610174 2395 2395 2395 93208803 25.9 0 10 745 3727.9 0 0 56031854/5603185 2386 2386 2386 92657740 26.9 0 MSG: reloading logging configuration 10 738 3695.5 0 0 56493451/5649345 2365 2365 2365 92716778 27.9 0 10 745 3727.8 0 0 56018249/5601824 2391 2391 2391 91939268 28.9 0 10 756 3784.0 0 0 56111387/5611138 2434 2434 2435 92760617 29.9 0 10 751 3763.2 0 0 56909299/5690929 2415 2415 2415 94589589 30.9 0 10 743 3720.4 0 0 56743451/5674345 2393 2393 2394 94007744 31.9 0 10 740 3701.1 0 0 56634460/5663446 2382 2382 2381 94029298 32.9 0 10 742 3713.7 0 0 55980456/5598045 2387 2387 2386 92359589 33.9 0 10 741 3705.5 0 0 56371550/5637155 2374 2374 2374 93070649 34.9 0 10 745 3729.4 0 0 56542132/5654213 2384 2384 2384 93323043 35.9 0 10 746 3731.4 0 0 56133536/5613353 2389 2389 2389 93059291 36.9 0 10 747 3738.9 0 0 55737497/5573749 2388 2388 2388 92461255 37.9 0 10 752 3763.1 0 0 55103102/5510310 2408 2408 2408 91561596 38.9 0 10 750 3753.3 0 0 55480099/5548009 2402 2402 2403 92163571 39.9 0 con new ops 1B lat bytes/per svrs new ops total time err 10 745 3731.6 0 0 56133000/5613300 2384 2384 2384 93322882 40.9 0 10 747 3738.7 0 0 55835431/5583543 2386 2386 2387 92600827 41.9 0 10 741 3713.2 0 0 54669004/5466900 2376 2376 2376 90439750 42.9 0 10 739 3701.9 0 0 54144761/5414476 2359 2359 2359 89544247 43.9 0 10 737 3687.7 0 0 53854332/5385433 2357 2357 2357 88714467 44.9 0 10 732 3665.2 0 0 54499541/5449954 2348 2348 2348 90060117 45.9 0 10 729 3649.6 0 0 54318027/5431802 2329 2329 2329 88905261 46.9 0 10 728 3646.8 0 0 54750580/5475058 2331 2331 2331 89516688 47.9 0 10 728 3645.1 0 0 54980749/5498074 2324 2324 2324 89975664 48.9 0 10 725 3630.7 0 0 56413670/5641367 2329 2329 2329 93476227 49.9 0 10 726 3636.9 0 0 55936710/5593671 2347 2347 2347 92509059 50.9 0 10 726 3633.5 0 0 55897549/5589754 2352 2352 2352 92984785 51.9 0 10 726 3632.6 0 0 55519381/5551938 2354 2354 2354 92128985 52.9 0 10 728 3643.3 0 0 55945915/5594591 2355 2355 2355 92625345 53.9 0 10 723 3621.2 0 0 55818057/5581805 2340 2340 2340 92013123 54.9 0 10 723 3619.8 0 0 56264557/5626455 2336 2336 2336 93062238 55.9 0 10 732 3661.2 0 0 55980990/5598099 2374 2374 2373 93104477 56.9 0 MSG: reloading logging configuration 10 732 3666.8 0 0 56241955/5624195 2365 2365 2365 93070508 57.9 0 10 732 3664.8 0 0 55710433/5571043 2365 2365 2365 91905169 58.9 0 10 735 3680.1 0 0 55864904/5586490 2364 2364 2363 92086367 59.9 0 con new ops 1B lat bytes/per svrs new ops total time err 10 744 3723.1 0 0 56150248/5615024 2380 2380 2380 92697563 60.9 0 10 595 2978.4 0 0 44920198/4492019 1904 1904 1904 74158050 60.9 0 MSG: checking for crashes ... ./functions: line 177: 28815 Terminated ( tsexec traffic_cop --stdout > $log ) ./functions: line 177: 29084 Terminated reload MSG: shutting down ... Success: test-log-refcounting test-multicert-loading test-multicert-loading test-privilege-elevation test-multicert-loading test-privilege-elevation --> Starting test: test-server-intercept MSG: bootstrapped Traffic Server into /tmp/test-server-intercept.14213 PREFIX: /tmp/test-server-intercept.14213 BINDIR: /opt/jenkins/tsqa-master/bin SYSCONFDIR: /tmp/test-server-intercept.14213/etc/trafficserver LIBDIR: /tmp/test-server-intercept.14213/lib LOGDIR: /tmp/test-server-intercept.14213/var/log/trafficserver RUNTIMEDIR: /tmp/test-server-intercept.14213/var/trafficserver PLUGINDIR: /opt/jenkins/tsqa-master/libexec/trafficserver INCLUDEDIR: /tmp/test-server-intercept.14213/include SNAPSHOTDIR: /tmp/test-server-intercept.14213/etc/trafficserver/snapshots records.config: /tmp/test-server-intercept.14213/etc/trafficserver/records.config remap.config: /tmp/test-server-intercept.14213/etc/trafficserver/remap.config plugin.config: /tmp/test-server-intercept.14213/etc/trafficserver/plugin.config ssl_multicert.config: /tmp/test-server-intercept.14213/etc/trafficserver/ssl_multicert.config storage.config: /tmp/test-server-intercept.14213/etc/trafficserver/storage.config hosting.config: /tmp/test-server-intercept.14213/etc/trafficserver/hosting.config volume.config: /tmp/test-server-intercept.14213/etc/trafficserver/volume.config ip_allow.config: /tmp/test-server-intercept.14213/etc/trafficserver/ip_allow.config MSG: cop is alive MSG: manager is alive MSG: server is alive MSG: waiting 1s gethostbyname(jtest.trafficserver.apache.org): Unknown host con new ops 1B lat bytes/per svrs new ops total time err 10 39 189.0 1 1 2938191/293819 176 176 176 5601550 0.7 0 10 193 964.2 1 1 14662659/1466265 741 741 741 25385184 1.6 0 10 318 1588.8 1 1 22852839/2285283 1102 1102 1102 37889796 2.6 0 10 409 2043.6 1 1 29745174/2974517 1346 1346 1346 48441251 3.6 0 10 484 2423.9 1 1 35393632/3539363 1558 1558 1558 57017337 4.6 0 10 543 2716.9 1 1 40449108/4044910 1718 1718 1718 65516597 5.6 0 10 592 2963.9 1 1 43873943/4387394 1891 1891 1891 71313825 6.7 0 10 631 3157.5 1 1 46637357/4663735 2078 2078 2078 76948972 7.6 0 10 654 3275.0 1 1 50112445/5011244 2156 2156 2155 82764539 8.6 0 10 684 3421.6 1 1 52113595/5211359 2211 2211 2211 85962509 9.6 0 10 712 3561.7 1 1 53506979/5350697 2261 2261 2261 87586211 10.6 0 10 726 3635.0 1 1 54750005/5475000 2285 2285 2284 89332154 11.6 0 10 741 3711.8 1 1 54985688/5498568 2359 2359 2359 90502241 12.6 0 10 751 3758.4 1 1 55164907/5516490 2438 2438 2438 91786922 13.6 0 10 758 3793.7 1 1 55053772/5505377 2470 2470 2470 91401103 14.6 0 10 769 3848.8 1 1 55461624/5546162 2494 2494 2494 92097350 15.6 0 10 773 3871.6 1 1 57312209/5731220 2473 2473 2473 94739422 16.7 0 10 774 3875.9 1 1 57767469/5776746 2446 2446 2446 94352592 17.6 0 10 778 3892.3 1 1 58477355/5847735 2476 2476 2476 96110526 18.6 0 10 782 3912.7 1 1 57989972/5798997 2527 2527 2527 94971112 19.6 0 con new ops 1B lat bytes/per svrs new ops total time err 10 780 3904.3 1 1 57599261/5759926 2534 2534 2534 94378824 20.6 0 10 772 3862.7 1 1 57780520/5778052 2500 2500 2500 94574563 21.6 0 10 768 3846.7 1 1 57380435/5738043 2464 2464 2464 93325167 22.6 0 10 774 3875.4 1 1 57146841/5714684 2455 2455 2455 92858812 23.6 0 10 765 3827.5 1 1 57553555/5755355 2424 2424 2424 94730185 24.6 0 10 749 3748.2 1 1 56302575/5630257 2406 2406 2406 93341575 25.6 0 10 753 3768.0 1 1 56642201/5664220 2438 2438 2439 93998644 26.6 0 10 765 3828.4 1 1 56525374/5652537 2471 2471 2472 93382898 27.6 0 10 770 3852.9 1 1 57910133/5791013 2463 2463 2463 95594436 28.6 0 10 777 3890.7 1 1 57104755/5710475 2471 2471 2471 94194861 29.6 0 10 775 3880.4 1 1 57097049/5709704 2458 2458 2458 93287683 30.6 0 10 776 3886.5 1 1 56832366/5683236 2480 2480 2480 93135893 31.6 0 10 773 3868.8 1 1 58002961/5800296 2494 2494 2494 95340813 32.6 0 10 774 3873.4 1 1 58333564/5833356 2496 2496 2496 95759603 33.6 0 10 779 3899.4 1 1 58666371/5866637 2509 2509 2509 96216966 34.6 0 10 784 3921.9 1 1 58846836/5884683 2492 2492 2492 96438460 35.6 0 10 792 3960.9 1 1 58749813/5874981 2519 2519 2519 96333408 36.6 0 10 783 3921.3 1 1 58830057/5883005 2516 2516 2516 96791970 37.6 0 10 784 3920.9 1 1 59359718/5935971 2534 2534 2535 98080033 38.6 0 10 782 3913.1 1 1 59250536/5925053 2532 2532 2532 97562676 39.6 0 con new ops 1B lat bytes/per svrs new ops total time err 10 781 3905.9 1 1 58940216/5894021 2513 2513 2514 96729295 40.6 0 10 783 3919.1 1 1 58818959/5881895 2499 2499 2499 95830994 41.6 0 10 784 3922.7 1 1 59424912/5942491 2488 2488 2489 97077844 42.6 0 10 781 3907.9 1 1 58446462/5844646 2502 2502 2502 95792759 43.6 0 10 778 3896.2 1 1 58604546/5860454 2515 2515 2515 96656768 44.6 0 10 776 3883.5 1 1 57876578/5787657 2523 2523 2523 95569207 45.6 0 10 774 3872.8 1 1 57533739/5753373 2507 2507 2507 94663161 46.6 0 10 780 3903.5 1 1 57468683/5746868 2499 2499 2499 94185721 47.6 0 10 779 3898.2 1 1 58334322/5833432 2488 2488 2488 95398583 48.6 0 10 782 3910.7 1 1 58056035/5805603 2495 2495 2495 94676506 49.6 0 10 776 3882.8 1 1 57602785/5760278 2488 2488 2488 94438779 50.6 0 10 778 3893.4 1 1 56902704/5690270 2507 2507 2507 93688223 51.6 0 10 771 3856.3 1 1 57638132/5763813 2485 2485 2485 94686400 52.6 0 10 773 3867.1 1 1 57559361/5755936 2480 2480 2480 94289141 53.6 0 10 766 3834.1 1 1 58090132/5809013 2456 2456 2456 95437427 54.6 0 10 768 3845.4 1 1 58257828/5825782 2455 2455 2455 95566723 55.6 0 10 773 3869.8 1 1 57644849/5764484 2466 2466 2466 94431417 56.6 0 10 771 3857.0 1 1 58278212/5827821 2465 2465 2464 95490439 57.6 0 10 774 3874.2 1 1 58033352/5803335 2472 2472 2472 95052762 58.6 0 10 774 3876.2 1 1 58546532/5854653 2467 2467 2466 96180571 59.6 0 con new ops 1B lat bytes/per svrs new ops total time err 10 780 3903.1 1 1 57990857/5799085 2470 2470 2470 94411116 60.6 0 10 624 3122.5 1 1 46392685/4639268 1976 1976 1976 75528892 60.6 0 MSG: checking for crashes ... ./functions: line 177: 8755 Terminated ( tsexec traffic_cop --stdout > $log ) MSG: shutting down ... Success: test-server-intercept test-multicert-loading test-privilege-elevation --> Starting test: test-ssl-certificates MSG: bootstrapped Traffic Server into /tmp/test-ssl-certificates.10317 PREFIX: /tmp/test-ssl-certificates.10317 BINDIR: /opt/jenkins/tsqa-master/bin SYSCONFDIR: /tmp/test-ssl-certificates.10317/etc/trafficserver LIBDIR: /tmp/test-ssl-certificates.10317/lib LOGDIR: /tmp/test-ssl-certificates.10317/var/log/trafficserver RUNTIMEDIR: /tmp/test-ssl-certificates.10317/var/trafficserver PLUGINDIR: /opt/jenkins/tsqa-master/libexec/trafficserver INCLUDEDIR: /tmp/test-ssl-certificates.10317/include SNAPSHOTDIR: /tmp/test-ssl-certificates.10317/etc/trafficserver/snapshots records.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/records.config remap.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/remap.config plugin.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/plugin.config ssl_multicert.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/ssl_multicert.config storage.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/storage.config hosting.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/hosting.config volume.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/volume.config ip_allow.config: /tmp/test-ssl-certificates.10317/etc/trafficserver/ip_allow.config MSG: cop is alive MSG: manager is alive MSG: server is alive Set proxy.config.http.server_ports, restart required Set proxy.config.diags.debug.tags MSG: waiting 2s to restart with SSL ports enabled MSG: waiting 6s for traffic_server to restart MSG: checking for the *.tsqa.trafficserver.apache.org certificate ... FAIL: openssl check for *.tsqa.trafficserver.apache.org failed unable to load certificate 139646220752800:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE FAIL: received certificate CN "", expected "*.tsqa.trafficserver.apache.org" MSG: checking for the sni.tsqa.trafficserver.apache.org certificate ... FAIL: openssl check for sni.tsqa.trafficserver.apache.org failed unable to load certificate 140446490658720:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE FAIL: received certificate CN "", expected "sni.tsqa.trafficserver.apache.org" MSG: checking for the port.tsqa.trafficserver.apache.org certificate ... FAIL: openssl check for port.tsqa.trafficserver.apache.org failed unable to load certificate 140139840124832:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE FAIL: received certificate CN "", expected "port.tsqa.trafficserver.apache.org" MSG: checking for the address.tsqa.trafficserver.apache.org certificate ... FAIL: openssl check for address.tsqa.trafficserver.apache.org failed unable to load certificate 140360439977888:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE FAIL: received certificate CN "", expected "address.tsqa.trafficserver.apache.org" ./functions: line 177: 9147 Terminated ( tsexec traffic_cop --stdout > $log ) MSG: shutting down ... Failure: test-ssl-certificates test-multicert-loading test-privilege-elevation --> Starting test: test-trafficline-metrics MSG: bootstrapped Traffic Server into /tmp/test-trafficline-metrics.28388 PREFIX: /tmp/test-trafficline-metrics.28388 BINDIR: /opt/jenkins/tsqa-master/bin SYSCONFDIR: /tmp/test-trafficline-metrics.28388/etc/trafficserver LIBDIR: /tmp/test-trafficline-metrics.28388/lib LOGDIR: /tmp/test-trafficline-metrics.28388/var/log/trafficserver RUNTIMEDIR: /tmp/test-trafficline-metrics.28388/var/trafficserver PLUGINDIR: /opt/jenkins/tsqa-master/libexec/trafficserver INCLUDEDIR: /tmp/test-trafficline-metrics.28388/include SNAPSHOTDIR: /tmp/test-trafficline-metrics.28388/etc/trafficserver/snapshots records.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/records.config remap.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/remap.config plugin.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/plugin.config ssl_multicert.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/ssl_multicert.config storage.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/storage.config hosting.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/hosting.config volume.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/volume.config ip_allow.config: /tmp/test-trafficline-metrics.28388/etc/trafficserver/ip_allow.config MSG: cop is alive MSG: manager is alive MSG: server is alive MSG: waiting 1s ./functions: line 177: 9558 Terminated ( tsexec traffic_cop --stdout > $log ) MSG: shutting down ... Success: test-trafficline-metrics RESULT SUMMARY ============== test-bootstrap ...OK test-log-configuration ...OK test-log-refcounting ...OK test-server-intercept ...OK test-trafficline-metrics ...OK test-ssl-certificates ...FAIL Build step 'Execute shell' marked build as failure
