zrhoffman opened a new issue #4496: In CDN-in-a-Box, ORT fails in mid and edge containers URL: https://github.com/apache/trafficcontrol/issues/4496 <!-- ************ STOP!! ************ If this issue identifies a security vulnerability, DO NOT submit it! Instead, contact the Apache Software Foundation Security Team at [email protected] and follow the guidelines at https://www.apache.org/security/ regarding vulnerability disclosure. --> <!-- - For *SUPPORT QUESTIONS*, use the [Traffic Control slack channels](https://traffic-control-cdn.slack.com) or [Traffic Control mailing lists](http://trafficcontrol.apache.org/mailing_lists/). - Before submitting, please **SEARCH GITHUB** for a similar issue or PR. --> ## I'm submitting a ... <!-- (check all that apply with "[x]") --> <!--- security vulnerability (STOP!! - see above)--> - [x] bug report - [ ] new feature / enhancement request - [ ] improvement request (usability, performance, tech debt, etc.) - [ ] other <!--(Please do not submit support requests here - see above)--> ## Traffic Control components affected ... <!-- (check all that apply with "[x]") --> - [x] CDN in a Box - [ ] Documentation - [ ] Grove - [x] Traffic Control Client (Python) - [ ] Traffic Monitor - [ ] Traffic Ops - [ ] Traffic Ops ORT - [ ] Traffic Portal - [ ] Traffic Router - [ ] Traffic Stats - [ ] Traffic Vault - [ ] unknown ## Current behavior: <!-- Describe how the bug manifests / how the current features are insufficient. --> When CDN-in-a-Box starts, the `edge` container is stuck [waiting for `remap.config` to exist](https://github.com/apache/trafficcontrol/blob/7714637a9f/infrastructure/cdn-in-a-box/edge/run.sh#L85-L87). ORT fails in the `mid` container, too. As a result, delivery services in CDN-in-a-Box are currently unusable, as Traffic Monitor's attempts to gather `/_astats` from the cache servers results in "connection refused". Using master branch. ## Expected / new behavior: <!-- Describe what the behavior would be without the bug / how the feature would improve Traffic Control --> ATS should be running in the `edge` and `mid` containers. ## Minimal reproduction of the problem with instructions: <!-- If the current behavior is a bug or you can illustrate your feature request better with an example, please provide the *STEPS TO REPRODUCE* and include the applicable TC version. --> To get stack traces: ```shell script docker-compose exec edge /usr/local/bin/traffic_ops_ort -k --dispersion 0 SYNCDS ALL https://trafficops.infra.ciab.test:443 admin:twelve ``` To simply verify that the mid and edge are not working: ```shell script docker-compose exec mid curl -v video.demo1.mycdn.ciab.test ``` ## Anything else: <!-- e.g. stacktraces, related issues, suggestions how to fix --> <details> <summary>Running ORT using the command from the "Minimal reproduction" (click to expand)</summary> ```shell script [root@mid site-packages]# /usr/local/bin/traffic_ops_ort -k --dispersion 0 SYNCDS ALL https://trafficops.infra.ciab.test:443 admin:twelve Mon Mar 16 17:59:31 UTC 2020 INFO: 2020-03-16 17:59:31,526 line 118 in configuration.__init__: Distribution detected as: 'centos' INFO: 2020-03-16 17:59:31,527 line 121 in configuration.__init__: Hostname detected as: 'mid' INFO: 2020-03-16 17:59:31,527 line 129 in configuration.__init__: ATS root installation directory set to '/' DEBUG: 2020-03-16 17:59:31,529 line 959 in connectionpool._new_conn: Starting new HTTPS connection (1): trafficops.infra.ciab.test:443 /usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'trafficops.infra.ciab.test'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, DEBUG: 2020-03-16 17:59:31,560 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "HEAD / HTTP/1.1" 404 0 WARNING: 2020-03-16 17:59:31,575 line 65 in utils.log_with_debug_info: restapi.py:__init__:210: Certificate verification warnings are disabled. DEBUG: 2020-03-16 17:59:31,575 line 65 in utils.log_with_debug_info: restapi.py:__init__:213: RestApiSession instance 0x7f39bcadfb38 initialized: Details: {'_session': None, '_host_ip': 'trafficops.infra.ciab.test', '_host_port': 443, '_api_version': '1.4', '_api_base_path': 'api/1.4/', '_ssl': True, '_headers': {'Content-Type': 'application/json; charset=UTF-8'}, '_verify_cert': False, '_create_session': False, '_max_retries': 5, '_RestApiSession__api_version_format_name': 'api_version', '_RestApiSession__api_version_format_value': '{api_version}', '_server_url': 'https://trafficops.infra.ciab.test:443/', '_api_base_url': 'https://trafficops.infra.ciab.test:443/api/1.4/'} DEBUG: 2020-03-16 17:59:31,575 line 65 in utils.log_with_debug_info: tosession.py:__init__:137: TOSession instance 0x7f39bcadfb38 initialized: Details: {'_session': None, '_host_ip': 'trafficops.infra.ciab.test', '_host_port': 443, '_api_version': '1.4', '_api_base_path': 'api/1.4/', '_ssl': True, '_headers': {'Content-Type': 'application/json; charset=UTF-8'}, '_verify_cert': False, '_create_session': False, '_max_retries': 5, '_RestApiSession__api_version_format_name': 'api_version', '_RestApiSession__api_version_format_value': '{api_version}', '_server_url': 'https://trafficops.infra.ciab.test:443/', '_api_base_url': 'https://trafficops.infra.ciab.test:443/api/1.4/', '_logged_in': False} INFO: 2020-03-16 17:59:31,576 line 62 in to_api.__init__: login attempt #0 INFO: 2020-03-16 17:59:31,576 line 150 in tosession.login: Connecting to Traffic Ops at https://trafficops.infra.ciab.test:443/... DEBUG: 2020-03-16 17:59:31,576 line 229 in retry.from_int: Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None) DEBUG: 2020-03-16 17:59:31,576 line 229 in retry.from_int: Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None) DEBUG: 2020-03-16 17:59:31,577 line 65 in utils.log_with_debug_info: restapi.py:create:254: Created internal requests Session instance 0x7f39bc612da0 INFO: 2020-03-16 17:59:31,577 line 155 in tosession.login: Connected. Authenticating... DEBUG: 2020-03-16 17:59:31,577 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:330: URL parameters are: [{'data': {'u': 'admin', 'p': 'twelve'}}] DEBUG: 2020-03-16 17:59:31,577 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:355: Built end-point to return: https://trafficops.infra.ciab.test:443/api/1.4/user/login DEBUG: 2020-03-16 17:59:31,578 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:401: Call parameters: {'headers': {'Content-Type': 'application/json; charset=UTF-8'}, 'verify': False, 'data': '{"u": "admin", "p": "twelve"}'} DEBUG: 2020-03-16 17:59:31,579 line 959 in connectionpool._new_conn: Starting new HTTPS connection (1): trafficops.infra.ciab.test:443 DEBUG: 2020-03-16 17:59:31,881 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "POST /api/1.4/user/login HTTP/1.1" 200 85 DEBUG: 2020-03-16 17:59:31,882 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:407: Response status: 200 OK INFO: 2020-03-16 17:59:31,882 line 179 in tosession.login: Authenticated. DEBUG: 2020-03-16 17:59:31,899 line 48 in services.<module>: Host system does NOT have systemd - stack trace: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/services.py", line 46, in <module> output = subprocess.check_output(["systemctl", "--no-pager"], stderr=subprocess.STDOUT) File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib64/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['systemctl', '--no-pager']' returned non-zero exit status 1. Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 294, in run from . import services File "<frozen importlib._bootstrap>", line 1023, in _handle_fromlist File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/services.py", line 48, in <module> logging.debug("Host system does NOT have systemd - stack trace:", exc_info=True,stack_info=True) INFO: 2020-03-16 17:59:31,900 line 47 in main_routines.syncDSState: starting syncDS state fetch INFO: 2020-03-16 17:59:31,900 line 249 in to_api.getMyUpdateStatus: Fetching update status from Traffic Ops DEBUG: 2020-03-16 17:59:31,901 line 65 in utils.log_with_debug_info: restapi.py:method_wrapper:118: Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',) DEBUG: 2020-03-16 17:59:31,901 line 258 in to_api.getMyUpdateStatus: update status fetch failure: OperationError("Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',)",) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 254, in getMyUpdateStatus r = self.get_server_update_status(server_name=self.hostname) File "/usr/local/lib/python3.6/site-packages/trafficops/restapi.py", line 119, in method_wrapper raise OperationError(msg) trafficops.restapi.OperationError: Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',) Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 316, in run updateRequired = syncDSState(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 49, in syncDSState updateStatus = conf.api.getMyUpdateStatus()[0] File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 258, in getMyUpdateStatus logging.debug("update status fetch failure: %r", e, exc_info=True, stack_info=True) DEBUG: 2020-03-16 17:59:31,902 line 65 in utils.log_with_debug_info: restapi.py:method_wrapper:118: Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',) DEBUG: 2020-03-16 17:59:31,902 line 258 in to_api.getMyUpdateStatus: update status fetch failure: OperationError("Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',)",) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 254, in getMyUpdateStatus r = self.get_server_update_status(server_name=self.hostname) File "/usr/local/lib/python3.6/site-packages/trafficops/restapi.py", line 119, in method_wrapper raise OperationError(msg) trafficops.restapi.OperationError: Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',) Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 316, in run updateRequired = syncDSState(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 49, in syncDSState updateStatus = conf.api.getMyUpdateStatus()[0] File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 258, in getMyUpdateStatus logging.debug("update status fetch failure: %r", e, exc_info=True, stack_info=True) DEBUG: 2020-03-16 17:59:31,903 line 65 in utils.log_with_debug_info: restapi.py:method_wrapper:118: Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',) DEBUG: 2020-03-16 17:59:31,903 line 258 in to_api.getMyUpdateStatus: update status fetch failure: OperationError("Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',)",) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 254, in getMyUpdateStatus r = self.get_server_update_status(server_name=self.hostname) File "/usr/local/lib/python3.6/site-packages/trafficops/restapi.py", line 119, in method_wrapper raise OperationError(msg) trafficops.restapi.OperationError: Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',) Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 316, in run updateRequired = syncDSState(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 49, in syncDSState updateStatus = conf.api.getMyUpdateStatus()[0] File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 258, in getMyUpdateStatus logging.debug("update status fetch failure: %r", e, exc_info=True, stack_info=True) CRITICAL: 2020-03-16 17:59:31,903 line 318 in main_routines.run: Server configuration unreachable, or not found in Traffic Ops! ERROR: 2020-03-16 17:59:31,903 line 319 in main_routines.run: Failed to fetch update status - connection was lost DEBUG: 2020-03-16 17:59:31,903 line 320 in main_routines.run: ConnectionError('Failed to fetch update status - connection was lost',) Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 316, in run updateRequired = syncDSState(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 49, in syncDSState updateStatus = conf.api.getMyUpdateStatus()[0] File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 260, in getMyUpdateStatus raise ConnectionError("Failed to fetch update status - connection was lost") ConnectionError: Failed to fetch update status - connection was lost Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 320, in run logging.debug("%r", e, exc_info=True, stack_info=True) DEBUG: 2020-03-16 17:59:31,905 line 65 in utils.log_with_debug_info: restapi.py:close:271: Internal requests Session instance 0x7f39bc612da0 closed and cleaned up ``` </details> Relevant line is ``` DEBUG: 2020-03-16 17:59:31,903 line 258 in to_api.getMyUpdateStatus: update status fetch failure: OperationError("Method [get_server_update_status] is not supported by this client's API version [1.4]; Supported versions: ('2.0',)",) ``` Support for API version 1.4 for `get_server_update_status` was removed in #4413. <details> <summary>Changing [the API version used](https://github.com/apache/trafficcontrol/blob/eb9cf79ea8/infrastructure/cdn-in-a-box/ort/traffic_ops_ort/to_api.py#L45) from <code>"1.4"</code> to <code>"2.0"</code> and re-running (click to expand)</summary> ```shell script [root@mid site-packages]# /usr/local/bin/traffic_ops_ort -k --dispersion 0 SYNCDS ALL https://trafficops.infra.ciab.test:443 admin:twelve Mon Mar 16 18:00:13 UTC 2020 INFO: 2020-03-16 18:00:13,164 line 118 in configuration.__init__: Distribution detected as: 'centos' INFO: 2020-03-16 18:00:13,165 line 121 in configuration.__init__: Hostname detected as: 'mid' INFO: 2020-03-16 18:00:13,165 line 129 in configuration.__init__: ATS root installation directory set to '/' DEBUG: 2020-03-16 18:00:13,166 line 959 in connectionpool._new_conn: Starting new HTTPS connection (1): trafficops.infra.ciab.test:443 /usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'trafficops.infra.ciab.test'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, DEBUG: 2020-03-16 18:00:13,197 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "HEAD / HTTP/1.1" 404 0 WARNING: 2020-03-16 18:00:13,212 line 65 in utils.log_with_debug_info: restapi.py:__init__:210: Certificate verification warnings are disabled. DEBUG: 2020-03-16 18:00:13,212 line 65 in utils.log_with_debug_info: restapi.py:__init__:213: RestApiSession instance 0x7fa27d1fa160 initialized: Details: {'_session': None, '_host_ip': 'trafficops.infra.ciab.test', '_host_port': 443, '_api_version': '2.0', '_api_base_path': 'api/2.0/', '_ssl': True, '_headers': {'Content-Type': 'application/json; charset=UTF-8'}, '_verify_cert': False, '_create_session': False, '_max_retries': 5, '_RestApiSession__api_version_format_name': 'api_version', '_RestApiSession__api_version_format_value': '{api_version}', '_server_url': 'https://trafficops.infra.ciab.test:443/', '_api_base_url': 'https://trafficops.infra.ciab.test:443/api/2.0/'} DEBUG: 2020-03-16 18:00:13,213 line 65 in utils.log_with_debug_info: tosession.py:__init__:137: TOSession instance 0x7fa27d1fa160 initialized: Details: {'_session': None, '_host_ip': 'trafficops.infra.ciab.test', '_host_port': 443, '_api_version': '2.0', '_api_base_path': 'api/2.0/', '_ssl': True, '_headers': {'Content-Type': 'application/json; charset=UTF-8'}, '_verify_cert': False, '_create_session': False, '_max_retries': 5, '_RestApiSession__api_version_format_name': 'api_version', '_RestApiSession__api_version_format_value': '{api_version}', '_server_url': 'https://trafficops.infra.ciab.test:443/', '_api_base_url': 'https://trafficops.infra.ciab.test:443/api/2.0/', '_logged_in': False} INFO: 2020-03-16 18:00:13,213 line 62 in to_api.__init__: login attempt #0 INFO: 2020-03-16 18:00:13,213 line 150 in tosession.login: Connecting to Traffic Ops at https://trafficops.infra.ciab.test:443/... DEBUG: 2020-03-16 18:00:13,213 line 229 in retry.from_int: Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None) DEBUG: 2020-03-16 18:00:13,214 line 229 in retry.from_int: Converted retries value: 5 -> Retry(total=5, connect=None, read=None, redirect=None, status=None) DEBUG: 2020-03-16 18:00:13,214 line 65 in utils.log_with_debug_info: restapi.py:create:254: Created internal requests Session instance 0x7fa278cb1898 INFO: 2020-03-16 18:00:13,214 line 155 in tosession.login: Connected. Authenticating... DEBUG: 2020-03-16 18:00:13,214 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:330: URL parameters are: [{'data': {'u': 'admin', 'p': 'twelve'}}] DEBUG: 2020-03-16 18:00:13,215 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:355: Built end-point to return: https://trafficops.infra.ciab.test:443/api/2.0/user/login DEBUG: 2020-03-16 18:00:13,215 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:401: Call parameters: {'headers': {'Content-Type': 'application/json; charset=UTF-8'}, 'verify': False, 'data': '{"u": "admin", "p": "twelve"}'} DEBUG: 2020-03-16 18:00:13,216 line 959 in connectionpool._new_conn: Starting new HTTPS connection (1): trafficops.infra.ciab.test:443 DEBUG: 2020-03-16 18:00:13,511 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "POST /api/2.0/user/login HTTP/1.1" 200 85 DEBUG: 2020-03-16 18:00:13,512 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:407: Response status: 200 OK INFO: 2020-03-16 18:00:13,512 line 179 in tosession.login: Authenticated. DEBUG: 2020-03-16 18:00:13,529 line 48 in services.<module>: Host system does NOT have systemd - stack trace: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/services.py", line 46, in <module> output = subprocess.check_output(["systemctl", "--no-pager"], stderr=subprocess.STDOUT) File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib64/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['systemctl', '--no-pager']' returned non-zero exit status 1. Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 294, in run from . import services File "<frozen importlib._bootstrap>", line 1023, in _handle_fromlist File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/services.py", line 48, in <module> logging.debug("Host system does NOT have systemd - stack trace:", exc_info=True,stack_info=True) INFO: 2020-03-16 18:00:13,530 line 47 in main_routines.syncDSState: starting syncDS state fetch INFO: 2020-03-16 18:00:13,530 line 249 in to_api.getMyUpdateStatus: Fetching update status from Traffic Ops DEBUG: 2020-03-16 18:00:13,531 line 65 in utils.log_with_debug_info: restapi.py:method_wrapper:110: Calling method [get] with keyword arguments [{'server_name': 'mid'}] via API endpoint method [get_server_update_status] DEBUG: 2020-03-16 18:00:13,532 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:330: URL parameters are: [{'server_name': 'mid'}] DEBUG: 2020-03-16 18:00:13,532 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:355: Built end-point to return: https://trafficops.infra.ciab.test:443/api/2.0/servers/mid/update_status DEBUG: 2020-03-16 18:00:13,533 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:401: Call parameters: {'headers': {'Content-Type': 'application/json; charset=UTF-8'}, 'verify': False} DEBUG: 2020-03-16 18:00:13,539 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "GET /api/2.0/servers/mid/update_status HTTP/1.1" 200 130 DEBUG: 2020-03-16 18:00:13,540 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:407: Response status: 200 OK DEBUG: 2020-03-16 18:00:13,540 line 262 in to_api.getMyUpdateStatus: Raw response from Traffic Ops: [{"host_name":"mid","upd_pending":true,"reval_pending":false,"use_reval_pending":true,"host_id":10,"status":"REPORTED","parent_pending":false,"parent_reval_pending":false}] DEBUG: 2020-03-16 18:00:13,540 line 51 in main_routines.syncDSState: Retrieved raw update status: Munch({'host_name': 'mid', 'upd_pending': True, 'reval_pending': False, 'use_reval_pending': True, 'host_id': 10, 'status': 'REPORTED', 'parent_pending': False, 'parent_reval_pending': False}) INFO: 2020-03-16 18:00:13,540 line 136 in main_routines.setStatusFile: Setting status file INFO: 2020-03-16 18:00:13,540 line 274 in to_api.getMyStatus: Fetching server status from Traffic Ops DEBUG: 2020-03-16 18:00:13,541 line 65 in utils.log_with_debug_info: restapi.py:method_wrapper:110: Calling method [get] with keyword arguments [{'query_params': {'hostName': 'mid'}}] via API endpoint method [get_servers] DEBUG: 2020-03-16 18:00:13,541 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:330: URL parameters are: [{}] DEBUG: 2020-03-16 18:00:13,541 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:337: URL query parameters are: [?hostName=mid] DEBUG: 2020-03-16 18:00:13,542 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:355: Built end-point to return: https://trafficops.infra.ciab.test:443/api/2.0/servers?hostName=mid DEBUG: 2020-03-16 18:00:13,542 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:401: Call parameters: {'headers': {'Content-Type': 'application/json; charset=UTF-8'}, 'verify': False} DEBUG: 2020-03-16 18:00:13,548 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "GET /api/2.0/servers?hostName=mid HTTP/1.1" 200 523 DEBUG: 2020-03-16 18:00:13,549 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:407: Response status: 200 OK DEBUG: 2020-03-16 18:00:13,550 line 288 in to_api.getMyStatus: Raw response from Traffic Ops: {"response":[{"cachegroup":"CDN_in_a_Box_Mid","cachegroupId":6,"cdnId":2,"cdnName":"CDN-in-a-Box","domainName":"infra.ciab.test","guid":null,"hostName":"mid","httpsPort":443,"id":10,"iloIpAddress":"","iloIpGateway":"","iloIpNetmask":"","iloPassword":"","iloUsername":"","interfaceMtu":1500,"interfaceName":"eth0","ip6Address":"fc01:9400:1000:8::2","ip6Gateway":"fc01:9400:1000:8::1","ipAddress":"172.16.239.2","ipGateway":"172.16.239.1","ipNetmask":"255.255.255.0","lastUpdated":"2020-03-16 17:14:40+00","mgmtIpAddress":"","mgmtIpGateway":"","mgmtIpNetmask":"","offlineReason":"","physLocation":"Apachecon North America 2018","physLocationId":1,"profile":"ATS_MID_TIER_CACHE","profileDesc":"Mid Cache - Apache Traffic Server","profileId":10,"rack":"","revalPending":false,"routerHostName":"","routerPortName":"","status":"REPORTED","statusId":3,"tcpPort":80,"type":"MID","typeId":12,"updPending":true,"xmppId":"mid","xmppPasswd":""}]} WARNING: 2020-03-16 18:00:13,551 line 145 in main_routines.setStatusFile: status directory does not exist, creating... INFO: 2020-03-16 18:00:13,551 line 171 in main_routines.setStatusFile: File '/opt/ort/status/REPORTED' to be created INFO: 2020-03-16 18:00:13,552 line 330 in main_routines.run: Processing Packages... INFO: 2020-03-16 18:00:13,552 line 112 in to_api.getMyPackages: Fetching this server's package list from Traffic Ops DEBUG: 2020-03-16 18:00:13,552 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:330: URL parameters are: [{}] DEBUG: 2020-03-16 18:00:13,552 line 65 in utils.log_with_debug_info: restapi.py:_build_endpoint:355: Built end-point to return: https://trafficops.infra.ciab.test:443/ort/mid/packages DEBUG: 2020-03-16 18:00:13,553 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:401: Call parameters: {'headers': {'Content-Type': 'application/json; charset=UTF-8'}, 'verify': False} DEBUG: 2020-03-16 18:00:13,624 line 437 in connectionpool._make_request: https://trafficops.infra.ciab.test:443 "GET /ort/mid/packages HTTP/1.1" 200 22 DEBUG: 2020-03-16 18:00:13,625 line 65 in utils.log_with_debug_info: restapi.py:_do_operation:407: Response status: 200 OK DEBUG: 2020-03-16 18:00:13,625 line 131 in to_api.getMyPackages: Raw package response: [] INFO: 2020-03-16 18:00:13,626 line 336 in main_routines.run: Done. INFO: 2020-03-16 18:00:13,626 line 338 in main_routines.run: Processing Services... WARNING: 2020-03-16 18:00:13,626 line 217 in main_routines.processServices: This system doesn't have systemd, services cannot be enabled/disabled INFO: 2020-03-16 18:00:13,626 line 344 in main_routines.run: Done. INFO: 2020-03-16 18:00:13,626 line 348 in main_routines.run: Processing Configuration Files... INFO: 2020-03-16 18:00:13,627 line 377 in config_files.initBackupDir: Initializing backup dir /opt/ort/backups INFO: 2020-03-16 18:00:13,627 line 150 in to_api.getMyConfigFiles: Fetching list of configuration files from Traffic Ops DEBUG: 2020-03-16 18:00:13,628 line 65 in utils.log_with_debug_info: restapi.py:close:271: Internal requests Session instance 0x7fa278cb1898 closed and cleaned up ERROR: 2020-03-16 18:00:13,628 line 2278 in tosession.__exit__: 'API' object has no attribute 'get_server_config_files' DEBUG: 2020-03-16 18:00:13,629 line 2279 in tosession.__exit__: <class 'AttributeError'> Stack (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/trafficops/tosession.py", line 2279, in __exit__ logging.debug("%s", exc_type, stack_info=traceback) Traceback (most recent call last): File "/usr/local/bin/traffic_ops_ort", line 11, in <module> load_entry_point('traffic-ops-ort==0.1.0', 'console_scripts', 'traffic_ops_ort')() File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 526, in main return doMain(args) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/__init__.py", line 368, in doMain return main_routines.run(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 349, in run if not processConfigurationFiles(conf): File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/main_routines.py", line 254, in processConfigurationFiles myFiles = conf.api.getMyConfigFiles(conf) File "/usr/local/lib/python3.6/site-packages/traffic_ops_ort/to_api.py", line 155, in getMyConfigFiles myFiles = self.get_server_config_files(host_name=self.hostname) AttributeError: 'API' object has no attribute 'get_server_config_files' ``` </details> Relevant line is ```shell script AttributeError: 'API' object has no attribute 'get_server_config_files' ``` The `get_server_config_files` endpoint method was removed in #4413. <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
