Re: Observations on flexus4 fastCache MOESI protocol

[Jason Zebchuk]

Hi,

I'm not going to directly address all of the points in your email, but I 
have a few comments that you might find helpful.

First, you are right that the InclusiveMOESI protocol won't really work 
for an L1 cache. The protocol was only intended to work only for an L2 
cache in a 2-level hierarchy.

If you have private L1 and private L2 caches, you should use the 
InclusiveMESI protocol for the L1 and the InclusiveMOESI protocol for 
the L2.  This restricts the "Owned" state to only existing in the L2 
cache. This combination will resolve many of the problems you pointed 
out in your email.

If you have a shared lower level cache, I would recommend using 
InclusiveMESI protocol for all levels of private caches. But note that 
3-level cache hierarchies have not been extensively tested with these 
protocols and they were not designed with such deep hierarchies in mind.

In regards to NAW accesses, there is code to support such accesses 
because it is required for the simulator to function, but the code is 
basically an after-thought and very little effort was put into making 
sure it was correct or sensible.

Finally, I'll agree there are some interesting behaviours in places. The 
code also makes some hidden assumptions about what is below it and how 
it behaves.  Specifically, the code was written to work with a 
FastCMPDirectory component above it, and there are some subtle 
dependencies between the protocol and the directory behaviour.


Jason



On 11-01-27 9:55 AM, Aristides Efthymiou wrote:
Hi All,

Looking at the details of the provided inclusive MOESI protocol found in
components/FastCache/InclusiveMOESI.hpp, I think I have found a couple of minor 
bugs and there's some possibly strange behaviour with downgrade and return 
snoop requests.

Others may use it as a baseline for experiments, so I'd like to ask the list 
subscriber's opinions about my observations below.
I'm assuming a general case where a core has 2 (or more) levels of dedicated caches and I 
am considering the core is at the "top".

1. Functionality
There is no StoreAccess (store from the core) in the action declarations.
  This means the model is suitable only for L2 and lower caches.
Moreover, the first action, reading a Modified block, would not work for an L1 
as the Dirty bit is lost to the cache above, which isn’t there, so we won't 
know that the block is dirty anymore.
Am I right about this?

2. Bugs in handling NonAllocatingWrites
DECLARE_REQUEST_ACTION( kModified,      kNAWAccess,     SendNone,       
NoAllocate, E_State, UpdateLRU, FillDirty,      HitNAWModified );
DECLARE_REQUEST_ACTION( kOwned,         kNAWAccess,     SendUpgrade,    
NoAllocate, E_State,    UpdateLRU,      FillWritable,   MissNAWOwned );
DECLARE_REQUEST_ACTION( kShared,        kNAWAccess,     SendUpgrade,    
NoAllocate, E_State,    UpdateLRU,      FillWritable,   MissNAWShared );

This is a non-allocating access, so the next state should stay/change to 
Modified as the block remains at this cache level.
Also the responses should be NAW-acks not Fills, though this shouldn't affect 
correctness.


3. Interesting/strange behaviour.
DECLARE_SNOOP_ACTION(   kOwned,         MM::ReturnReq,  MM::ReturnReplyDirty,   
NoSnoop,        S_State,        HitReturnReqOwned, false, false );
DECLARE_SNOOP_ACTION(   kOwned,         MM::Invalidate, MM::InvalidateAck,      
DoSnoop,        I_State,        HitInvalidateOwned, false, true );
Interesting difference for the response to snoop commands for a block in Owned 
state.
Invalidate, responds with just an Ack and ReturnReq responds with Dirty. The 
Dirty response seems strange; there’s another cache with a copy in the system.
There’s also an issue with forwarding the snoop up or not. See next comment.

DECLARE_SNOOP_ACTION(   kModified,      MM::Downgrade,  MM::DownUpdateAck,      
DoSnoop,        O_State,                HitDowngradeModified, ......... );
DECLARE_SNOOP_ACTION(   kExclusive,     MM::Downgrade,  SnoopDepend,            
DoSnoop,        DependState,    HitDowngradeExclusive,  .......  );
Two issues:
- For Modified, why pass Downgrade snoop message up? Any copies above a 
Modified block can be read-only.  There’s nothing to downgrade and, actually, 
looking at the action for a Shared (i.e. read-only) block  for a downgrade 
request, it gives poison!
- Exclusive seems ok but consider this case: A block is Modified in L1 and 
Exclusive in L2.
   When a Downgrade request is received in L2, it is forwarded to L1. L1 
changes state to Owned and responds with DownUpdateAck. L2 also changes to 
Owned and passes on the same response.
   So we are left with 2 levels having Owned as their state. That is OK in 
general, except for handling ReturnReq (see previous comment) where snooping is 
not forwarded up.


DECLARE_SNOOP_ACTION(   kExclusive,     MM::ReturnReq,  MM::ReturnReply,        
DoSnoop,        DependState,HitReturnReqExclusive, ........ );
Two issues:
-  Dirty status is not passed down when above this Exclusive block there is a 
Modified one. The point here is that if we have a single cache with a modified 
block, we get the Dirty status in snoop response, but if we have a hierarchy of 
caches with the top one holding a modified block (and the others exclusive), we 
don’t get the dirty status. Does this matter?
- In the method code for this action, the snoop message sent up the hierarchy 
is Downgrade.
   Depending on the snoop reply the next state could be Owned (reply is 
UpdateAck) or Shared (reply is Ack). There is a subtle difference of this and 
the plain Downgrade snoop request: The top (originally) modified cache changes 
to Owned for downgrade as do all the ones below it which were exclusive. For a 
return request, the top changes to Shared and the ones below to Owned. Does it 
matter?

Thanks,

Aris

PS:  Since this is my first post, I'm also sending regards all to the people  I 
met at EPFL during my visit there in November 2009. I hope you are well.