So definitely not a memory problem. Do your client requests (and sql requests) just timeout after a bit? If I'm reading your netstat output correctly, traffic_server just has 2600 sockets open for port 80, which does not seem like too many.
It is quite easy in a single client stress test to run out of ephemeral ports from the single client to the server. But that should affect connections to other processes such as SQL. I assume that you've already adjusted the local port range and other port reuse settings on your tests boxes to set up more ephemeral ports. On Mon, Oct 19, 2020 at 10:51 AM pentester <[email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hello Susan! > > Thank you for your feedback. Well, i think my conclusion is wrong about > 500 concurrent connections against ATS. > Because: httpress -n 100000000 -c 500 -t 4 'http://mytestsite.domain/' > > 4 threads x 500 concurrent connections against ATS does = 2000 > concurrent connnects. > > Anyways, it gets down for me: the symtoms is: under that high load > test: ATS starves all available sockets on server. > So as result: ATS = is unable to accept new connections, MYSQL, HTTPD > (origin) also gets down (mysql server has gone aways etc). > Sockets starvation + CPU starvation. > > In fact when i do stress test from other box (over WAN) any request > reaching ATS should get (and gets i can confirm) 403 (ATS's 403) > because i wrote simple rule to get always 403 against requests. So i > can confirm my concurrent requests never reaches origin (HTTPD). > > > Here is what i see under the stress-test: > > # traffic_top > > Disk Used 1.2M Ram > Hit 40.8% GET 1.2% 200 53.5% > Disk Total > 837.8M Fresh 0.3% HEAD 0.2% 206 0.0% > Ram > Used 99.8K Revalidate 0.0% POST 0.2% 301 > 0.0% > Ram > Total 4.3G Cold 98.8% 2xx 53.5% 302 > 1.3% > Lookups 7.2K Changed 0.5% 3xx 45.4% 304 > 44.0% > Writes 0.0 Not > Cache 0.0% 4xx 1.1% 404 0.8% > Updates 37.0 No > Cache 0.0% 5xx 0.0% 502 0.0% > Deletes 0.0 Fresh (ms) 44.1G Conn Fail 0.0 100 > B 45.2% > Read Activ 0.0 Reval (ms) 0.0 Other Err 78.0K 1 > KB 16.2% > Writes Act 1.0 Cold (ms) 161.7M Abort 22.0 3 > KB 1.1% > Update Act 0.0 Chang (ms) 29.0G 5 > KB 4.9% > Entries 42.0 Not (ms) 337.8G 10 > KB 7.9% > Avg Size 29.3K No (ms) 0.0 1 > MB 21.2% > DNS Lookup 7.2K DNS Hit 100.0% > 1 > MB 0.2% > DNS Hits 7.2K DNS Entry 3.0 > CLIENT ORIGIN SERVER > Requests 7.2K Head Bytes 16.1M Requests 7.2K Head > Bytes 1.9M > Req/Conn 0.4 Body Bytes 266.4M Req/Conn 1.2 Body > Bytes 176.8M > New Conn 19.7K Avg Size 39.0K New Conn 5.8K Avg > Size 24.8K > Curr Conn 921.0 Net (bits) 2.3G Curr Conn 1.0 Net > (bits) 1.4G > Active Con 144.0 Resp (ms) 96.6 > Dynamic KA 5.0 > xxxxxxxxxx (r)esponse (q)uit (h)elp > (a)bsolute > > > - ------------------------- > > > > $ watch -n 2 "netstat -tulpan|grep -i "traffic_ser"|wc -l" > - ------------- > Every 2.0s: netstat -tulpan|grep -i traffic_ser|wc -l > Mon Oct 19 17:35:22 2020 > > 914 > > - ------------- > > > > > # free -m > total used free shared buff/cache a > vailable > Mem: 7809 1488 5831 163 490 > 5919 > Swap: 0 0 0 > > > - ---------------- > > # top -u nobody > top - 17:46:09 up 1 day, 10:42, 1 user, load average: 4.18, 2.35, > 1.83 > Tasks: 155 total, 3 running, 152 sleeping, 0 stopped, 0 zombie > %Cpu(s): 95.6 us, 3.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.7 hi, 0.5 > si, 0.0 st > MiB Mem : 7809.8 total, 5843.2 free, 1473.2 used, 493.4 > buff/cache > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5934.3 avail > Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 167102 nobody 20 0 1491900 90612 14648 S 362.1 1.1 31:59.49 > [TS_MAIN] > > 43697 nobody 20 0 329940 19284 8572 S 0.0 0.2 0:35.26 > traffic_manager > > > > - ------------ > > # netstat -tulpan|grep -i :80|wc -l > 2599 > > > - ------------ > > > > > > > Here is my rule: > > - ------------------ > > // config_1.conf > > cond %{READ_REQUEST_HDR_HOOK} > cond %{CLIENT-HEADER:CF-Connecting-IP} /MY_.IP._FROM_.WHERE_.I_.DoS/ > set-status 403 > > - ------------------- > > > I'm not using cache mechanism and not doing negative caching etc. This > is a simple 403 page of ATS where i hit always. > > Regarding RAM = allocated memory 4 GB. > > > > Here is my records.config (a bit messed up while playing with it :)) > > > > $ cat /usr/local/etc/trafficserver/records.config > ####################################################################### > ####### > # *NOTE*: All options covered in this file should be documented in the > docs: > # > # https://docs.trafficserver.apache.org/records.config > ####################################################################### > ####### > > ####################################################################### > ####### > # Thread configurations. Docs: > # > https://docs.trafficserver.apache.org/records.config#thread-variables > ####################################################################### > ####### > CONFIG proxy.config.exec_thread.autoconfig INT 1 > CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.5 > CONFIG proxy.config.exec_thread.limit INT 2 > CONFIG proxy.config.accept_threads INT 1 > CONFIG proxy.config.task_threads INT 2 > CONFIG proxy.config.cache.threads_per_disk INT 8 > CONFIG proxy.config.exec_thread.affinity INT 1 > > ####################################################################### > ####### > # Specify server addresses and ports to bind for HTTP and HTTPS. Docs: > # > > https://docs.trafficserver.apache.org/records.config#proxy.config.http.server_ports > ####################################################################### > ####### > CONFIG proxy.config.http.server_ports STRING 80 8096:ipv6 > > ####################################################################### > ####### > # Via: headers. Docs: > # > > https://docs.trafficserver.apache.org/records.config#proxy-config-http-insert-response-via-str > ####################################################################### > ####### > CONFIG proxy.config.http.insert_request_via_str INT 0 > CONFIG proxy.config.http.insert_response_via_str INT 0 > #CONFIG proxy.config.http.response_via_str 0 > #CONFIG proxy.config.http.insert_age_in_response > #CONFIG proxy.config.http.response_via_str GoogleWebServer > CONFIG proxy.config.http.response_server_enabled INT 0 > ####################################################################### > ####### > # Parent proxy configuration, in addition to these settings also see > parent.config. Docs: > # > > https://docs.trafficserver.apache.org/records.config#parent-proxy-configuration > # > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/parent.config.en.html > ####################################################################### > ####### > CONFIG proxy.config.http.parent_proxy_routing_enable INT 0 > CONFIG proxy.config.http.parent_proxy.retry_time INT 300 > CONFIG proxy.config.http.parent_proxy.connect_attempts_timeout INT 30 > CONFIG proxy.config.http.forward.proxy_auth_to_parent INT 0 > CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 1 > CONFIG proxy.config.http.cache.ignore_client_no_cache INT 1 > > ####################################################################### > ####### > # HTTP connection timeouts (secs). Docs: > # > > https://docs.trafficserver.apache.org/records.config#http-connection-timeouts > ####################################################################### > ####### > CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 20 > CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 20 > CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 20 > CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 20 > CONFIG proxy.config.http.transaction_active_timeout_in INT 900 > CONFIG proxy.config.http.transaction_active_timeout_out INT 0 > CONFIG proxy.config.http.accept_no_activity_timeout INT 120 > CONFIG proxy.config.net.default_inactivity_timeout INT 86400 > > ####################################################################### > ####### > # Origin server connect attempts. Docs: > # > > https://docs.trafficserver.apache.org/records.config#origin-server-connect-attempts > ####################################################################### > ####### > CONFIG proxy.config.http.connect_attempts_max_retries INT 10 > CONFIG proxy.config.http.connect_attempts_max_retries_dead_server INT 3 > CONFIG proxy.config.http.connect_attempts_rr_retries INT 3 > CONFIG proxy.config.http.connect_attempts_timeout INT 30 > CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800 > CONFIG proxy.config.http.down_server.cache_time INT 60 > CONFIG proxy.config.http.down_server.abort_threshold INT 10 > > ####################################################################### > ####### > # Negative response caching, for redirects and errors. Docs: > # > > https://docs.trafficserver.apache.org/records.config#negative-response-caching > ####################################################################### > ####### > CONFIG proxy.config.http.negative_caching_enabled INT 0 > CONFIG proxy.config.http.negative_caching_lifetime INT 1800 > > ####################################################################### > ####### > # Proxy users variables. Docs: > # > https://docs.trafficserver.apache.org/records.config#proxy-user-variables > ####################################################################### > ####### > CONFIG proxy.config.http.insert_client_ip INT 1 > CONFIG proxy.config.http.insert_squid_x_forwarded_for INT 1 > > ####################################################################### > ####### > # Security. Docs: > # https://docs.trafficserver.apache.org/records.config#security > ####################################################################### > ####### > CONFIG proxy.config.http.push_method_enabled INT 0 > > ####################################################################### > ####### > # Enable / disable HTTP caching. Useful for testing, but also as an > # overridable (per remap) config > ####################################################################### > ####### > CONFIG proxy.config.http.cache.http INT 1 > > ####################################################################### > ####### > # Cache control. Docs: > # https://docs.trafficserver.apache.org/records.config#cache-control > # > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/cache.config.en.html > ####################################################################### > ####### > CONFIG proxy.config.http.cache.ignore_client_cc_max_age INT 1 > CONFIG proxy.config.http.normalize_ae INT 1 > CONFIG proxy.config.http.cache.cache_responses_to_cookies INT 1 > CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT 1 > # > > https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-when-to-revalidate > CONFIG proxy.config.http.cache.when_to_revalidate INT 0 > # > > https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-required-headers > CONFIG proxy.config.http.cache.required_headers INT 2 > > ####################################################################### > ####### > # Heuristic cache expiration. Docs: > # > https://docs.trafficserver.apache.org/records.config#heuristic-expiration > ####################################################################### > ####### > CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 3600 > CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 > CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.10 > > ####################################################################### > ####### > # Network. Docs: > # https://docs.trafficserver.apache.org/records.config#network > ####################################################################### > ####### > CONFIG proxy.config.net.connections_throttle INT 50000 > CONFIG proxy.config.net.max_connections_in INT 50000 > CONFIG proxy.config.net.max_connections_active_in INT 50000 > > ####################################################################### > ####### > # RAM and disk cache configurations. Docs: > # https://docs.trafficserver.apache.org/records.config#ram-cache > # > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/storage.config.en.html > ####################################################################### > ####### > CONFIG proxy.config.cache.ram_cache.size INT -1 > CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304 > # > > https://docs.trafficserver.apache.org/records.config#proxy-config-cache-limits-http-max-alts > CONFIG proxy.config.cache.limits.http.max_alts INT 5 > # > > https://docs.trafficserver.apache.org/records.config#proxy-config-cache-max-doc-size > CONFIG proxy.config.cache.max_doc_size INT 0 > CONFIG proxy.config.cache.min_average_object_size INT 8000 > > ####################################################################### > ####### > # Logging Config. Docs: > # > https://docs.trafficserver.apache.org/records.config#logging-configuration > # > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/logging.yaml.en.html > ####################################################################### > ####### > CONFIG proxy.config.log.logging_enabled INT 3 > CONFIG proxy.config.log.max_space_mb_for_logs INT 25000 > CONFIG proxy.config.log.max_space_mb_headroom INT 1000 > CONFIG proxy.config.log.rolling_enabled INT 1 > CONFIG proxy.config.log.rolling_interval_sec INT 86400 > CONFIG proxy.config.log.rolling_size_mb INT 10 > CONFIG proxy.config.log.auto_delete_rolled_files INT 1 > CONFIG proxy.config.log.periodic_tasks_interval INT 5 > > ####################################################################### > ####### > # These settings control remapping, and if the proxy allows (open) > forward proxy or not. Docs: > # > https://docs.trafficserver.apache.org/records.config#url-remap-rules > # > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/remap.config.en.html > ####################################################################### > ####### > CONFIG proxy.config.url_remap.remap_required INT 1 > # > > https://docs.trafficserver.apache.org/records.config#proxy-config-url-remap-pristine-host-hdr > CONFIG proxy.config.url_remap.pristine_host_hdr INT 0 > # > https://docs.trafficserver.apache.org/records.config#reverse-proxy > CONFIG proxy.config.reverse_proxy.enabled INT 1 > > ####################################################################### > ####### > # SSL Termination. Docs: > # > > https://docs.trafficserver.apache.org/records.config#client-related-configuration > # > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/ssl_multicert.config.en.html > ####################################################################### > ####### > CONFIG proxy.config.ssl.client.verify.server INT 0 > CONFIG proxy.config.ssl.client.CA.cert.filename STRING NULL > CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA-AES256- > GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM- > SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS- > AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM- > SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA- > AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA- > AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA- > AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS- > AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128- > SHA:DHE-DSS-AES128- > SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3- > SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA > > ####################################################################### > ####### > # Debugging. Docs: > # > > https://docs.trafficserver.apache.org/records.config#diagnostic-logging-configuration > ####################################################################### > ####### > CONFIG proxy.config.diags.debug.enabled INT 0 > CONFIG proxy.config.diags.debug.tags STRING http|dns > # ToDo: Undocumented > CONFIG proxy.config.dump_mem_info_frequency INT 0 > CONFIG proxy.config.http.slow.log.threshold INT 0 > > CONFIG proxy.config.hostdb.host_file.path STRING /etc/hosts > > > #CONFIG proxy.config.thread.default.stacksize INT 3145728 > > > CONFIG proxy.config.cache.ram_cache.size INT 4G > CONFIG proxy.config.net.inactivity_check_frequency INT 80 > > > > - ---------------- > > > > Cheers, > > > > > On Mon, 2020-10-19 at 08:29 -0500, Susan Hinrichs wrote: > > Sounds like ATS should be able to keep up with that work > > load. However, > > I've not worked with httpress specifically, so I'm not certain what > > kind of > > RPS those arguments would correspond to.. > > > > How did your ATS box fail? Did it crash? Did it just get so slow > > that the > > requests started timing out? Is your request mapping through to an > > origin? Or hitting cache? > > > > I would start looking at memory usage. If ATS starts swapping, > > things go > > downhill fast. If you are caching, how much ram cache do you have > > configured? If you aren't caching, does you configuration still have > > ram > > cache enabled? > > > > Susan > > > > On Sun, Oct 18, 2020 at 3:11 PM pentester <[email protected]> > > wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > Hash: SHA1 > > > > > > Hello Folks! > > > > > > I need your valuable advise regarding how to fine-tune ATS for > > > highly > > > loaded server. > > > > > > The situation: > > > Test server: (VPS) CPU(s) 4 (Intel(R) Xeon(R) CPU E5-2630 v4 @ > > > 2.20GHz), Total RAM: 8 GB, HDD (SSD) > > > OS: Centos 8 > > > ATSv: 8.1.0 > > > > > > Problem: According to ATS documentation and it's default config > > > files > > > it can handle 30k concurrent requests. That's great! > > > > > > //records.config > > > CONFIG proxy.config.net.connections_throttle INT 30000 > > > CONFIG proxy.config.net.max_connections_in INT 30000 > > > CONFIG proxy.config.net.max_connections_active_in INT 30000 > > > > > > > > > > > > **During the the stress-test: 400-500 concurrent connections > > > against > > > simple 403 page completely downs ATS. (over WAN) > > > > > > httpress -n 100000000 -c 500 -t 4 'http://mytestsite.domain/' > > > > > > > > > (?) Question: > > > Should i play with settings / fine-tune them (if yes, please let me > > > know them) or better to upgrade HW / or any other advise on this > > > please. > > > > > > > > > Played a bit with the following settings (increased/decreased > > > them). No > > > avail. > > > > > > proxy.config.net.connections_throttle > > > proxy.config.net.max_connections_in > > > proxy.config.net.max_connections_active_in > > > > > > > > > proxy.config.http.keep_alive_no_activity_timeout_in > > > proxy.config.http.keep_alive_no_activity_timeout_out > > > proxy.config.http.transaction_no_activity_timeout_in > > > proxy.config.http.transaction_no_activity_timeout_out > > > > > > > > > > > > Thanks in advance! > > > > > > > > > > > > > > > > > > - -----BEGIN PGP PUBLIC KEY BLOCK----- > > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > > > mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH5 > > > c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFNr > > > hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9P > > > tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qrY > > > maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6s > > > 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63c > > > gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hFn > > > 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5IL > > > zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJF > > > va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0o > > > 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQAB > > > tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF5 > > > AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77G > > > Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJdaz > > > GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdWp > > > p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKPk > > > Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUVZ > > > WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eDW > > > RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfgb > > > AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIcoN > > > YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3n > > > hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCWw > > > bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUsU > > > WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANsd > > > iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwHO > > > 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPjs > > > 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN+ > > > /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj2 > > > YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRhZ > > > 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmujn > > > Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6d > > > FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xScY > > > k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66nn > > > sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/n > > > qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl68 > > > kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWlc > > > gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbAx > > > sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtWr > > > 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG7 > > > oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx9 > > > eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYOl > > > DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJl > > > DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uIP > > > U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFUY > > > a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9E > > > zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTbv > > > 0/I9kH7m8tCtYKI= > > > =vsDs > > > - -----END PGP PUBLIC KEY BLOCK----- > > > -----BEGIN PGP SIGNATURE----- > > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > > > iQIcBAEBAgAGBQJfjKGcAAoJEMrVQenMmzu+Sa0P/2BIU1biMZj/MM7oVIiNecl8 > > > GTfiTl2s+2Df56LA6o5IBKKBcKNJQHKwK2aOd9bhF5HY4uH9hl53Aq4mWcN8L3bY > > > 5ugmT9M7aSd5qRJvyWN1E6ryCLzaagdz+vSC63xi9G9slsdOZ7xVIgYE6nVLWOVG > > > 48zokAwh7MxjIkogdRCTOQrgehZWh32ugXgQoxXFSozge+jO0nFwJmdB52fNsgEA > > > tVENKE5xhYMdvGNTdRoq8RZ0PWJ0/2lhut77rnHWQ9eo/PRlMxEPgByWZkvLVUZ5 > > > lJKYpEWFVYrnox3D9bIl49I0bqNqWAVOfVMQs09cMRnSoqJxTPy/TMCOLNInp0mL > > > 4J+O65ga690Pap0ZeywSIDIIGy3eC14m9ACf8dW8JP7h75CpGhwZLtvWV/S+nkcG > > > +ZW6XNKoU4H5Jbi/PT9OZmYeJu0CsmlxjBRaq2oexlpEBt6pwaN7B347eb1DlCEr > > > 8dKUzpeik7PNTwkNCxq+tDHrQdbC1ANJyPN3avKvzmKAiAkn3DK+IPBAY84On0LH > > > 270iO96EAgRg53/csSCJJLxSnpBuJs8O1xGy6OFI4xxsDMBygSjo7AzNgu53tNDa > > > P6lQynfx+hLyYJu61CUY3yj6HAF5phlONz57ZkPt4BVu2P1RbSeILY7w9sacEDOR > > > I8G8prtCJ2SLFJXCvmOj > > > =7rQ7 > > > -----END PGP SIGNATURE----- > > > > > > > - -- > > > Hörmətlə, > > XRITDX KİMM (CERT.GOV.AZ) > > - ------------------------------ > > > > > > - -----BEGIN PGP PUBLIC KEY BLOCK----- > Version: GnuPG v2.0.22 (GNU/Linux) > > mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH5 > c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFNr > hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9P > tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qrY > maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6s > 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63c > gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hFn > 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5IL > zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJF > va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0o > 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQAB > tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF5 > AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77G > Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJdaz > GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdWp > p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKPk > Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUVZ > WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eDW > RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfgb > AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIcoN > YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3n > hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCWw > bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUsU > WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANsd > iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwHO > 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPjs > 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN+ > /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj2 > YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRhZ > 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmujn > Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6d > FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xScY > k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66nn > sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/n > qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl68 > kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWlc > gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbAx > sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtWr > 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG7 > oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx9 > eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYOl > DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJl > DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uIP > U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFUY > a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9E > zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTbv > 0/I9kH7m8tCtYKI= > =vsDs > - -----END PGP PUBLIC KEY BLOCK----- > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.22 (GNU/Linux) > > iQIcBAEBAgAGBQJfjbZcAAoJEMrVQenMmzu+0poQAOaUujECxax8x4wvmk11jbw6 > k8K80ryCXDmV5gnZj7DEcwpYrNEM49S2DSeHVUckQOngOVz5lsPFuGxdKimIuNbs > L7W/1AP11VPeyZOiJjwYZB6/S9xEsUGGxKzCz5dE1hkpQgai4NToFSeOXLOT/8m4 > 9nQOW2S+cvamTKi0NQV9leqGFr4lkGeFXG623YFsfT/+FsmZN0/aFQfuw3qkvZjE > liDOpEdzAEOr3vMrA6A56ijIgC/Z47t7bGwE/nNmcBLcKgh9qCq3dpg7eVyg6C1O > 5GpG9tZ5TlvswzvHd/waFoPBCZWEIgtv6qeIOIuMzibua+HEDIEbd8cbQQDzAmyM > ib9Y/9Fk/O2vFD7wpJX5epbttvIyGlUftsgDt7CjaaHXaU/kbVLKmkw/gGhPW6iC > GJcMosxWUCsDQ7Uh3k8g1BWDNEl2Q6UPYgVokcyDEToac+yXbZkBbJHCYM0tJe7y > c22pKdGAJ4LkLZYhZfjQ30FwA/z+6MjGHbUbjBnk/lJ0DAyV0i0t+aL8VYP+CrOM > vz6O0IWXW0iK8Bpj6/FpuVMrQRWAKimbnHj2VO8sWQbl3WcQCFK7MA8Td/B2yV+t > cOl0M+/eo9eG8aQXU/zq5GPt8sj77qda2XjHPud/GnWXJBtqEKV+GZwB/AQ3yBn7 > AsUvUTOnsrAy7FSKduMN > =Ckx4 > -----END PGP SIGNATURE----- > >
