Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-04 Thread Henrik Nordström
mån 2012-12-03 klockan 22:25 -0700 skrev Alex Rousskov: I disagree with this logic. Yes, sawActivity approach supports long chains, but they are not the loop's fault! If some chain is too long, the bug is in that chain's code and should be fixed there. If it cannot be fixed there, then the

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-04 Thread Henrik Nordström
mån 2012-12-03 klockan 23:12 -0700 skrev Alex Rousskov: BTW, once zero-delay events are removed from the code, the event queue will no longer need to be a part of the wasActivity loop and the loop will disappear. Note: Legacy zero-delay addEvent events are meant to be called once per comm

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-04 Thread Alex Rousskov
On 12/04/2012 01:47 AM, Henrik Nordström wrote: mån 2012-12-03 klockan 22:25 -0700 skrev Alex Rousskov: I disagree with this logic. Yes, sawActivity approach supports long chains, but they are not the loop's fault! If some chain is too long, the bug is in that chain's code and should be

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-04 Thread Alex Rousskov
On 12/04/2012 01:59 AM, Henrik Nordström wrote: mån 2012-12-03 klockan 23:12 -0700 skrev Alex Rousskov: BTW, once zero-delay events are removed from the code, the event queue will no longer need to be a part of the wasActivity loop and the loop will disappear. Note: Legacy zero-delay

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-04 Thread Henrik Nordström
tis 2012-12-04 klockan 08:39 -0700 skrev Alex Rousskov: heavy events in the context of addEvent events are really this is very heavy, and there is only allowed to be one and only one heavy event per comm invocation. If there is multiple heavy jobs competing for time they are meant to get

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-04 Thread Henrik Nordström
tis 2012-12-04 klockan 08:39 -0700 skrev Alex Rousskov: There are several ways to interpret the designer intent when looking at undocumented code. I cannot say whether all of the currently remaining zero-delay events use your interpretation, but I am certain that I have added zero-delay

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Henrik Nordström
sön 2012-12-02 klockan 22:27 -0700 skrev Alex Rousskov: My understanding is that the bug is in Squid store rebuilding(?) code that assumes that Squid will have time to do some I/O before the next timed event is fired, but actually there is a lot of non-I/O work to do, In such case the bug is

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Alex Rousskov
On 12/03/2012 01:44 AM, Henrik Nordström wrote: The store rebuilding does eventAdd(storeRebuild, RebuildStep, rb, 0.01, 1); which says both delay = 0.01, avoid saturating things too much weight = 1, this is a heavy job, slow down event.cc processing when this is seen. Only

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Henrik Nordström
mån 2012-12-03 klockan 09:07 -0700 skrev Alex Rousskov: Yes. IIRC, it has been broken before my AsyncCall changes to the main loop. The sawActivity code is from the AsyncCall merge in rev 8812. This will poll non-comm event loops until they are completely idle having nothing to do.

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Alex Rousskov
On 12/03/2012 02:01 PM, Henrik Nordström wrote: mån 2012-12-03 klockan 09:07 -0700 skrev Alex Rousskov: I was going to agree with that wholeheartedly, but then I thought about signals. If we visit each non-waiting engine once, then we will only process signals once per main loop step. Is that

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Henrik Nordström
mån 2012-12-03 klockan 15:53 -0700 skrev Alex Rousskov: True (although the current small maximum select(2) delay is a result of bugs elsewhere in the code and should not really be there). If we can do the right thing here easily, we should (instead of adding more future problems). And I think

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Amos Jeffries
On 04.12.2012 05:07, Alex Rousskov wrote: On 12/03/2012 01:44 AM, Henrik Nordström wrote: The store rebuilding does eventAdd(storeRebuild, RebuildStep, rb, 0.01, 1); which says both delay = 0.01, avoid saturating things too much weight = 1, this is a heavy job, slow down event.cc

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Henrik Nordström
mån 2012-12-03 klockan 15:53 -0700 skrev Alex Rousskov: We just need to make sure that heavy events (certain timed events and signals) interrupt the sawActivity loop. This will fix the issue you are facing without introducing new problems. Another question on this.. how to fit the concept of

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Henrik Nordström
mån 2012-12-03 klockan 09:07 -0700 skrev Alex Rousskov: Could you please fix this properly by adding AsyncEngine::timeTillNextEvent() or similar API and removing loop_delay manipulation from checkEvents() as discussed above? The attached patch adds AsyncEngine::timeTillNextEvent() API. It

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Alex Rousskov
On 12/03/2012 04:16 PM, Henrik Nordström wrote: mån 2012-12-03 klockan 15:53 -0700 skrev Alex Rousskov: Ah, I see another problem with the each non-waiting engine runs once approach. One engine may create work for other engines or for itself, including async calls and run this now lightweight

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Alex Rousskov
On 12/03/2012 04:36 PM, Henrik Nordström wrote: mån 2012-12-03 klockan 15:53 -0700 skrev Alex Rousskov: We just need to make sure that heavy events (certain timed events and signals) interrupt the sawActivity loop. This will fix the issue you are facing without introducing new problems.

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-03 Thread Alex Rousskov
On 12/03/2012 04:18 PM, Amos Jeffries wrote: Would it make sense to do one of: * have the AsyncCall queue drain with a timeout. So that no sequence of calls can lag any of the sensitive engines too much. That along with the event engines scheduling their heavyweight actions as Calls instead

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-02 Thread Henrik Nordström
fre 2012-11-30 klockan 15:38 -0700 skrev Alex Rousskov: On 11/30/2012 02:25 PM, Henrik Nordström wrote: We should look into why it is at all needed. From what I can understand it should not be needed. Agreed. Please do that if you can. Found it. The event loop gets saturated with timed

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-12-02 Thread Alex Rousskov
On 12/02/2012 06:46 PM, Henrik Nordström wrote: fre 2012-11-30 klockan 15:38 -0700 skrev Alex Rousskov: On 11/30/2012 02:25 PM, Henrik Nordström wrote: We should look into why it is at all needed. From what I can understand it should not be needed. Agreed. Please do that if you can.

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-11-30 Thread Alex Rousskov
On 11/28/2012 04:07 PM, Henrik Nordström wrote: - each worker restarting at startup due to rock store DB open failures (timeout). This seems to settle once the rock store have completed rebuilding, but not 100% sure that's the cause as logs are a bit inclusive. Might also be related to the

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-11-30 Thread Henrik Nordström
fre 2012-11-30 klockan 10:47 -0700 skrev Alex Rousskov: IIRC, Rock store diskers should process queries while rebuilding so db open should not fail due to rebuild itself. I bet this is actually related to the problem discussed below. I think so too. - kid registration failures at startup

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-11-30 Thread Alex Rousskov
On 11/30/2012 02:25 PM, Henrik Nordström wrote: We should look into why it is at all needed. From what I can understand it should not be needed. Agreed. Please do that if you can. Have the foreground/background store rebuild logics been changed somehow? I do not recall changes to

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-11-29 Thread Amos Jeffries
On 29/11/2012 8:13 p.m., Henrik Nordström wrote: tor 2012-11-29 klockan 15:08 +1300 skrev Amos Jeffries: Does anybody know the rationale for having xmemset at all? I don't see muhc of a meaningful rationale. memory debugging tools already handle it without any wrapper. Done. Sliced away

Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load

2012-11-28 Thread Henrik Nordström
tor 2012-11-29 klockan 15:08 +1300 skrev Amos Jeffries: Does anybody know the rationale for having xmemset at all? I don't see muhc of a meaningful rationale. memory debugging tools already handle it without any wrapper. Regards Henrik