You must have some stack trace once all the retries are exhausted (tries=36, retries=35). can you please share that.
On Mon, Aug 7, 2017 at 11:53 PM, jeff saremi <[email protected]> wrote: > Just checked > > THe keys are URLs. The region name has that and also the intended lookup > key is a URL. So I think they are consistent > > > ________________________________ > From: jeff saremi <[email protected]> > Sent: Monday, August 7, 2017 11:18:52 AM > To: [email protected] > Subject: Re: Thrift servers not connecting to RegionServers > > > Not intended at all. > let me investigate that > > thanks Ted > > ________________________________ > From: Ted Yu <[email protected]> > Sent: Monday, August 7, 2017 10:52:10 AM > To: [email protected] > Subject: Re: Thrift servers not connecting to RegionServers > > The formation of row key (starting with http) looks to be different from > the region (which starts with table name). > > Is this expected ? > > On Mon, Aug 7, 2017 at 10:34 AM, jeff saremi <[email protected]> > wrote: > > > This happens so frequently to us and we still haven't figured out why > > > > We have this problem a few times a day where the Thrift server reports > > errors like: > > > > > > 2017-08-07 10:28:45,686 INFO [thrift-worker-54] > client.RpcRetryingCaller: > > Call exception, tries=31, retries=35, started=452454 ms ago, > > cancelled=false, msg=row 'http://anotherhost.com/ > > colonel-brewery-tours-lebanon.html' on table 'OurTable' at > > region=ClickStreamTable122016,http://somehost.com/,1501285630924. > > 9772ed631b4a7a3a76a9c8647c6f8273., hostname=co4aapa29412175, > 16020,1501881562692, > > seqNum=10880 > > > > > > and the regionserver in question (co4aapa29412175) shows no problems > > either on the status page or in its logs. > > > > Restarting Thrift servers are not helping either > > > > Does anyone have any ideas on what to check? > > thanks > > > > >
