Hi Robert, I can't be sure, but it sounds like the homestead component of Clearwater (the subscriber store) might not be running, or might have locked up.
We currently have an open issue (https://github.com/Metaswitch/crest/issues/162) that if NTP corrects the system time by a large amount, homestead may reject all requests sent to it. This can be cleared by rebooting the all-in-one node (log into the node and run `sudo reboot`). If that doesn't fix the problem the next thing I would try is restarting homestead-prov. Log into the node and run `sudo service homestead-prov stop`. This will stop the process, and monit (our monitoring layer) will automatically restart it. If that doesn't work, can you please send the homestead-prov.logs across? These are in /var/log/homestead-prov/homestead-prov*.log Also, can you please check for me that the homestead schema has installed correctly? You can check this by running "cqlsh -3", then "use homestead_cache;", and finally "describe tables;" - this should report the impu, impi and impi_mapping tables. If any of these steps fail (for example, because the homestead_cache keyspace is missing), then you'll need to recreate the keyspace. You can do this most simply by removing and reinstalling the homestead package. Thanks, Ellie -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Robert Chen Sent: 01 December 2014 05:27 To: [email protected] Subject: Re: [Clearwater] Ellis error: Failed to update the server Looks like the mailing list does not like the attachment. Here is the log: 2014-12-01 03:51:02,827 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.42ms 2014-12-01 03:51:12,550 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.72ms 2014-12-01 03:51:43,219 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.34ms 2014-12-01 03:52:03,389 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.33ms 2014-12-01 03:52:23,289 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:52:43,802 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:53:04,043 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:53:14,089 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.34ms 2014-12-01 03:53:24,181 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.46ms 2014-12-01 03:53:54,437 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:54:14,812 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.37ms 2014-12-01 03:54:34,967 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:54:55,173 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:55:05,397 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.34ms 2014-12-01 03:55:15,257 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:55:35,683 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:55:55,743 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.32ms 2014-12-01 03:56:16,007 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.33ms 2014-12-01 03:56:31,159 D static:80 25511:140048494843648] Checking if addressbook.html is allowed 2014-12-01 03:56:31,162 I web:1447 25511:140048494843648] 200 GET /addressbook.html (73.189.53.119) 3.06ms 2014-12-01 03:56:31,268 D static:80 25511:140048494843648] Checking if addressbook.html is allowed 2014-12-01 03:56:31,271 I web:1447 25511:140048494843648] 200 GET /addressbook.html (73.189.53.119) 2.94ms 2014-12-01 03:56:31,502 I web:1447 25511:140048494843648] 304 GET /gab (73.189.53.119) 2.43ms 2014-12-01 03:57:06,669 I web:1447 25511:140048494843648] 200 GET /ping (172.31.44.185) 0.33ms 2014-12-01 03:57:09,453 D static:80 25511:140048494843648] Checking if is allowed 2014-12-01 03:57:09,458 I web:1447 25511:140048494843648] 200 GET / (73.189. 53.119) 4.71ms 2014-12-01 03:57:09,625 D static:80 25511:140048494843648] Checking if is allowed 2014-12-01 03:57:09,630 I web:1447 25511:140048494843648] 200 GET / (73.189. 53.119) 5.18ms 2014-12-01 03:57:10,260 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,261 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,262 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,263 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,264 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,266 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,267 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,268 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,268 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,269 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,270 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,270 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 1 queued requests. 2014-12-01 03:57:10,271 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,271 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 2 queued requests. 2014-12-01 03:57:10,271 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,271 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 3 queued requests. 2014-12-01 03:57:10,272 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,272 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 4 queued requests. 2014-12-01 03:57:10,272 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,273 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 5 queued requests. 2014-12-01 03:57:10,273 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,273 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 6 queued requests. 2014-12-01 03:57:10,274 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,274 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 7 queued requests. 2014-12-01 03:57:10,274 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,274 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 8 queued requests. 2014-12-01 03:57:10,275 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,275 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 9 queued requests. 2014-12-01 03:57:10,275 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,275 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 10 queued requests. 2014-12-01 03:57:10,276 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,276 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 11 queued requests. 2014-12-01 03:57:10,276 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,277 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 12 queued requests. 2014-12-01 03:57:10,277 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,277 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 13 queued requests. 2014-12-01 03:57:10,277 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,278 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 14 queued requests. 2014-12-01 03:57:10,278 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,278 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 15 queued requests. 2014-12-01 03:57:10,279 W homestead:237 25511:140048494843648] Passing SIP password in the clear over http 2014-12-01 03:57:10,279 D simple_httpclient:107 25511:140048494843648] max_clients limit reached, request queued. 10 active, 16 queued requests. 2014-12-01 03:57:10,279 W iostream:507 25511:140048494843648] Connect error on fd 17: ECONNREFUSED 2014-12-01 03:57:10,279 W iostream:507 25511:140048494843648] Connect error on fd 18: ECONNREFUSED 2014-12-01 03:57:10,279 W iostream:507 25511:140048494843648] Connect error on fd 19: ECONNREFUSED 2014-12-01 03:57:10,280 W iostream:507 25511:140048494843648] Connect error on fd 20: ECONNREFUSED 2014-12-01 03:57:10,280 W iostream:507 25511:140048494843648] Connect error on fd 21: ECONNREFUSED 2014-12-01 03:57:10,280 W iostream:507 25511:140048494843648] Connect error on fd 22: ECONNREFUSED 2014-12-01 03:57:10,280 W iostream:507 25511:140048494843648] Connect error on fd 23: ECONNREFUSED 2014-12-01 03:57:10,281 W iostream:507 25511:140048494843648] Connect error on fd 24: ECONNREFUSED 2014-12-01 03:57:10,281 W iostream:507 25511:140048494843648] Connect error on fd 25: ECONNREFUSED 2014-12-01 03:57:10,281 W iostream:507 25511:140048494843648] Connect error on fd 26: ECONNREFUSED 2014-12-01 03:57:10,282 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.020529985427856445,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b16990>,effective_url='http://localhost:8889/public/sip%3A6505550019% 0x2b16990>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,282 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,283 E web:1447 25511:140048494843648] 502 GET /accounts/robertchen%40chinamobile.com/numbers/?cb=12d0c395f9V0 (73.189.53.119) 27.00ms 2014-12-01 03:57:10,283 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,283 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.021397113800048828,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b24e10>,effective_url='http://localhost:8889/public/sip%3A6505550078% 0x2b24e10>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,284 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,284 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,284 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,284 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.021451950073242188,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b241d0>,effective_url='http://localhost:8889/public/sip%3A6505550127% 0x2b241d0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,285 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,285 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,285 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,286 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.021663904190063477,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b24290>,effective_url='http://localhost:8889/public/sip%3A6505550168% 0x2b24290>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,286 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,286 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,286 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,311 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.021754980087280273,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b1c390>,effective_url='http://localhost:8889/public/sip%3A6505550249% 0x2b1c390>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,311 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,311 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,311 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,312 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.04528999328613281,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b16210>, effective_url='http://localhost:8889/public/sip%3A6505550321%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,312 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,312 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,312 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,313 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.04555702209472656,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b16550>, effective_url='http://localhost:8889/public/sip%3A6505550354%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,313 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,313 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,313 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,314 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.04580497741699219,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b16190>, effective_url='http://localhost:8889/public/sip%3A6505550402%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,314 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,314 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,314 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,315 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.04603719711303711,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b16790>, effective_url='http://localhost:8889/public/sip%3A6505550430%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,315 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,315 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,315 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,316 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.04622912406921387,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b160d0>, effective_url='http://localhost:8889/public/sip%3A6505550468%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,316 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,316 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,316 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,317 W iostream:507 25511:140048494843648] Connect error on fd 17: ECONNREFUSED 2014-12-01 03:57:10,317 W iostream:507 25511:140048494843648] Connect error on fd 18: ECONNREFUSED 2014-12-01 03:57:10,317 W iostream:507 25511:140048494843648] Connect error on fd 19: ECONNREFUSED 2014-12-01 03:57:10,317 W iostream:507 25511:140048494843648] Connect error on fd 20: ECONNREFUSED 2014-12-01 03:57:10,317 W iostream:507 25511:140048494843648] Connect error on fd 21: ECONNREFUSED 2014-12-01 03:57:10,318 W iostream:507 25511:140048494843648] Connect error on fd 22: ECONNREFUSED 2014-12-01 03:57:10,318 W iostream:507 25511:140048494843648] Connect error on fd 23: ECONNREFUSED 2014-12-01 03:57:10,318 W iostream:507 25511:140048494843648] Connect error on fd 24: ECONNREFUSED 2014-12-01 03:57:10,318 W iostream:507 25511:140048494843648] Connect error on fd 25: ECONNREFUSED 2014-12-01 03:57:10,318 W iostream:507 25511:140048494843648] Connect error on fd 26: ECONNREFUSED 2014-12-01 03:57:10,319 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.0373990535736084,buffer=None, _body=None,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b16dd0>,effective_url='http://localhost:8889/public/sip%3A6505550594% 0x2b16dd0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,319 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,319 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,320 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,320 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.03661990165710449,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29d10>, effective_url='http://localhost:8889/public/sip%3A6505550612%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,320 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,320 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,321 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,321 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.03674006462097168,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29ed0>, effective_url='http://localhost:8889/public/sip%3A6505550637%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,321 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,322 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,322 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,322 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.03672003746032715,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29a90>, effective_url='http://localhost:8889/public/sip%3A6505550671%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,322 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,323 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,323 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,323 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.03683805465698242,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29d50>, effective_url='http://localhost:8889/public/sip%3A6505550676%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,324 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,324 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,324 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,325 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.01263880729675293,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29e50>, effective_url='http://localhost:8889/public/sip%3A6505550714%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,325 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,325 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,325 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,325 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.01265406608581543,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29c10>, effective_url='http://localhost:8889/public/sip%3A6505550751%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,325 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,326 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,326 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,326 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.01233220100402832,buffer=None,_body=Non e,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29250>, effective_url='http://localhost:8889/public/sip%3A6505550764%40example.com/a ssociated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,326 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,326 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,326 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,327 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.011998891830444336,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b294d0>,effective_url='http://localhost:8889/public/sip%3A6505550829% 0x2b294d0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,327 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,327 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,327 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,327 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.011666059494018555,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29c90>,effective_url='http://localhost:8889/public/sip%3A6505550852% 0x2b29c90>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,328 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,328 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,328 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,328 W iostream:507 25511:140048494843648] Connect error on fd 17: ECONNREFUSED 2014-12-01 03:57:10,328 W iostream:507 25511:140048494843648] Connect error on fd 18: ECONNREFUSED 2014-12-01 03:57:10,328 W iostream:507 25511:140048494843648] Connect error on fd 19: ECONNREFUSED 2014-12-01 03:57:10,329 W iostream:507 25511:140048494843648] Connect error on fd 20: ECONNREFUSED 2014-12-01 03:57:10,329 W iostream:507 25511:140048494843648] Connect error on fd 21: ECONNREFUSED 2014-12-01 03:57:10,329 W iostream:507 25511:140048494843648] Connect error on fd 22: ECONNREFUSED 2014-12-01 03:57:10,329 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.010533809661865234,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b290d0>,effective_url='http://localhost:8889/public/sip%3A6505550886% 0x2b290d0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,329 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,330 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,330 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,330 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.0102081298828125,buffer=None, _body=None,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29bd0>,effective_url='http://localhost:8889/public/sip%3A6505550912% 0x2b29bd0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,330 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,330 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,330 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,331 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.009733915328979492,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29990>,effective_url='http://localhost:8889/public/sip%3A6505550915% 0x2b29990>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,331 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,331 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,331 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,331 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.009505987167358398,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29650>,effective_url='http://localhost:8889/public/sip%3A6505550925% 0x2b29650>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,332 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,332 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,332 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,332 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.009031057357788086,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b29ad0>,effective_url='http://localhost:8889/public/sip%3A6505550944% 0x2b29ad0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,332 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,333 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,333 D utils:89 25511:140048494843648] Still expecting 0 callbacks 2014-12-01 03:57:10,333 W utils:73 25511:140048494843648] Non-OK HTTP response. HTTPResponse(code=599,request_time=0.008651971817016602,buffer=None,_body=No ne,time_info={},request=<tornado.httpclient.HTTPRequest object at 0x2b292d0>,effective_url='http://localhost:8889/public/sip%3A6505550990% 0x2b292d0>40ex ample.com/associated_private_ids',headers={},error=HTTPError('HTTP 599: Connection closed',)) 2014-12-01 03:57:10,333 W numbers:113 25511:140048494843648] Failed to fetch private identities from homestead 2014-12-01 03:57:10,333 E web:718 25511:140048494843648] Cannot send error response after headers written 2014-12-01 03:57:10,333 D util -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Robert Chen Sent: 2014年11月30日 21:13 To: [email protected] Subject: [Clearwater] Ellis error: Failed to update the server I have been testing Clearwater on AWS AIO for a while, but lately, I got this error and cannot create or update the numbers. The tail of ellis-0.log is attached. Looks like ellis cannot get the private id from homestead. But I checked homestead and its database. All seems OK. Can someone give me some suggestions? Thanks. Robert Chen _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater _______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
