Fair enough, just a thought. On Wednesday, April 15, 2015, Nuria Ruiz <[email protected]> wrote:
> Given that batching code is been deployed since earlier (March16th) than > the 1st event listed by Marcel (April 9th) and since then we have swapped > the EL box (April 3rd/4th) we probably want to look at system issues. > > On my opinion it is probably easier to see with tcpdump whether inserts > are being sent, rather than trying to reply traffic to repro the problem. > > Thanks, > > Nuria > > > On Wed, Apr 15, 2015 at 5:14 PM, Dan Andreescu <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> >This sounds like the fixes we did last quarter to the batch insertion >>> basically hid the problem instead of making it go away. >>> I think we are mixing things here, when we had issues with batching code >>> we never saw a pattern of "no-events-whatsoever-in-any-table for an hour". >>> We saw events dropped in bursts here and there but certainly not an "hour >>> long blackout". >>> >>> Also, there were no events dropped when we did the major backfilling in >>> early march where the db sustained quite a bit of load as we had to insert >>> those one by one. >>> >>> So (while I am not saying we could not uncover a code issue in our end) >>> we have not seen this particular error pattern before. >>> >> >> I didn't mean to suggest we saw this error before. I was trying to say >> that intuitively the error seems very similar. That is, over time, the lag >> grows and at some point it's so big that we lose a bunch of data all at >> once. I was just saying that because the first place I'd look is at that >> change. For example, I'd try replicating by simulating the same traffic >> and then I'd revert to the original logic before batch inserts and try >> that. We've all looked at this code but we must be missing something big. >> >> _______________________________________________ >> Analytics mailing list >> [email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');> >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> >
_______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
