Mark Chu-Carroll created AURORA-513:
---------------------------------------
Summary: Errors during thrift RPC calls cause ugly stack dumps
Key: AURORA-513
URL: https://issues.apache.org/jira/browse/AURORA-513
Project: Aurora
Issue Type: Bug
Components: Client
Reporter: Mark Chu-Carroll
Assignee: Mark Chu-Carroll
Anonymized error message below. The user did an update; during the course of
the update, what appears to be a transient network glitch caused a thrift
socket to become None, which led to an invalid attempt to dereference None,
which caused a stack dump. The actual error is in thrift-generated code.
INFO] Examining instances: [360, 361, 362, 363, 364, 365]
INFO] Adding instances: [360, 361, 362, 363, 364, 365]
INFO] Instances added
INFO] Watching instances: [360, 361, 362, 363, 364, 365]
Traceback (most recent call last):
File "...aurora-client/twitter/common/app/application.py", line 738, in
_wrap_method
return_code = method()
File "...aurora-client/twitter/common/app/application.py", line 760, in
<lambda>
main = lambda: main_method(*args, **kwargs)
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/base.py",
line 72, in wrapped_function
return fn(*args)
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/commands/core.py",
line 614, in update
resp = api.update_job(config, options.health_check_interval_seconds,
options.shards)
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/hooks/hooked_api.py",
line 184, in update_job
config, health_check_interval_seconds=health_check_interval_seconds,
instances=instances))
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/hooks/hooked_api.py",
line 145, in _hooked_call
resp = api_call()
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/__init__.py",
line 128, in update_job
return updater.update(instances)
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/updater.py",
line 455, in update
if not self._update(instance_configs):
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/updater.py",
line 143, in _update
failed_instances = self._watcher.watch(instances_to_watch) if
instances_to_watch else set()
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/instance_watcher.py",
line 95, in watch
running_tasks = self._get_tasks_by_instance_id(instance_ids)
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/instance_watcher.py",
line 134, in _get_tasks_by_instance_id
resp = self._scheduler.getTasksStatus(query)
File
"...aurora-client/.deps/apache.aurora.clientv2-0.5.1_DEV1401741199-py2.6.egg/apache/aurora/client/api/scheduler_client.py",
line 264, in method_wrapper
return method(*(args + auth_args))
File
"...aurora-client/.deps/apache.gen.aurora-0.5.1_DEV1401741199-py2.6.egg/gen/apache/aurora/api/ReadOnlyScheduler.py",
line 129, in getTasksStatus
return self.recv_getTasksStatus()
File
"...aurora-client/.deps/apache.gen.aurora-0.5.1_DEV1401741199-py2.6.egg/gen/apache/aurora/api/ReadOnlyScheduler.py",
line 140, in recv_getTasksStatus
(fname, mtype, rseqid) = self._iprot.readMessageBegin()
File
"...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py",
line 317, in readMessageBegin
self.readJSONArrayStart()
File
"...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py",
line 305, in readJSONArrayStart
self.readJSONSyntaxChar(LBRACKET)
File
"...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py",
line 210, in readJSONSyntaxChar
current = self.reader.read()
File
"...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/protocol/TJSONProtocol.py",
line 139, in read
self.data = self.protocol.trans.read(1)
File
"...aurora-client/.deps/thrift-0.9.1-py2.6-macosx-10.4-x86_64.egg/thrift/transport/THttpClient.py",
line 97, in read
return self.__http.file.read(sz)
AttributeError: 'NoneType' object has no attribute 'read'
--
This message was sent by Atlassian JIRA
(v6.2#6252)
