Vladimir, Alexey, Let me describe my current vision of the task; please correct me if I'm wrong somewhere.
1. The only class adding and retrieving metadata to/from sys-cache is *CacheObjectBinaryProcessorImpl*. I expect all code managing discovery events flow to be placed here. Two the most important methods here are *addMeta(...)* and *metadata(...)*. I found out that both methods may be called during serialization and deserialization phases. *So at the moment I'm going to block all threads trying to do anything with not-yet-confirmed metadata*. I don't see any possibility to distinguish serialize call from deserialize call on BinaryProcessor level. If you know how to implement this more efficiently, please let me know. 2. I identified two use cases where I need to manage versioning of metadata. The first one is related to situation when cluster accepts new version of metadata and some client node still has older version of metadata. In that case client needs a version number on incoming binary object to compare it with version number of locally available metadata for this object and to request metadata if needed. The second case occurs when two changes to the same typeId are proposed by different nodes in cluster. In this situation it is possible that one change is already spread across the cluster when another one is still in progress. Acknowledge message for the first change should not unblock any threads waiting for the second one. The only way I see to achieve this is to add version number as described in initial email. I need some assistance here to think over a change of adding metadata version number to current binary object format. Thanks, Sergey. On Fri, Jan 13, 2017 at 12:28 PM, Alexey Goncharuk < alexey.goncha...@gmail.com> wrote: > Vladimir, > > Given that metadata updates will be initiated with DiscoveryCustomEvent, I > see no problems neither with total message ordering (the order will be the > same on all nodes and consistent with discovery events) nor with failover - > the event is marked with @TcpDiscoveryEnsureDelivery, so the event should > be fired even in the case of coordinator failure. > > I think handling metadata on clients might be tricky, however fore-fetch > approach is proven to work in the scope of IGNITE-4157. > > 2017-01-13 12:04 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>: > > > Sergey, > > > > In my understanding the only difficult thing here is how to disallow > usage > > of not-yet-confirmed metadata during serialization. As we assume metadata > > updates to be relatively rare, can we always initiate message send from > > coordinator node? In this case we will have total order of all metadata > > updates, what will make conflict resolution trivial. > > > > Alex, how do you think - will this approach work? I am concerned about > > coordinator failover. > > > > On Thu, Jan 12, 2017 at 4:09 PM, Sergey Chugunov < > > sergey.chugu...@gmail.com> > > wrote: > > > > > Hello Ignite devs, > > > > > > I came up with the following design proposal for IGNITE-4302 > > > <https://issues.apache.org/jira/browse/IGNITE-4302>, which is very > > similar > > > to described here > > > <http://apache-ignite-developers.2346864.n4.nabble. > > com/IGNITE-4157-design- > > > proposal-td11996.html> > > > proposal for IGNITE-4157 <https://issues.apache.org/ > > > jira/browse/IGNITE-4157>, > > > "*Use discovery custom messages instead of marshaller cache*". > > > > > > Protocol is very similar to IGNITE-4157 and based on exchanging two > > > messages: *MetadataProposed*/*MetadataAccepted*. > > > > > > 1. When a node wants to add or update existing metadata, it sends > > > *MetadataProposed* message with the update. > > > 2. Coordinator node checks for any conflicts with existing metadata > > for > > > this typeId. In case of no conflicts coordinator assigns a version > > > number > > > to new metadata and sends Proposed message further. > > > In case of conflict it simply marks this message as IN_CONFLICT and > > > sends it to the original node. > > > 3. All nodes upon receiving Proposed message update local metadata > > > including version number and put it to "pending_acceptance" status. > > [1] > > > 4. Coordinator acknowledges proposed message with *MetadataAccepted* > > > message, marking it with the version number from proposed message. > > > 5. Each node receiving accepted message checks local version number > > and > > > the one from the message. If they are equal, metadata is considered > as > > > accepted by the cluster. [2] > > > > > > New nodes joining the cluster receive metadata on discovery phase. > > > > > > As discovery messages are delivered to clients asynchronously, there > is a > > > possibility that cluster may consider and agree upon update to metadata > > > before client receives initial proposed message. > > > In that case I suggest to attach version number of metadata > BinaryObject > > > was serialized with to that BinaryObject so client may check whether it > > can > > > deserialize this object safely or need to request some updated version > of > > > metadata. > > > > > > [1] When a metadata is marked as "pending_acceptance" it should be > > > prohibited to use it to serialize binary objects but it still can be > used > > > for deserialization. > > > At the same time from code perspective I didn't find a clear > distinction > > > like "this method is used only for serialization and that method is > > called > > > only during deserialization". So for now the only way I see is to block > > all > > > operations for a given typeId until MetadataAccepted message for this > > > typeId is arrived. > > > > > > [2] If MetadataAccepted message has a version number less than > currently > > > pending, it is ignored and all threads waiting for this metadata to be > > > accepted remain blocked. > > > It may be possible for new joining node to receive a MetadataAccepted > > > message with version number greater than local copy, in that case it is > > > fine to apply metadata update and declare metadata accepted. > > > > > > > > > Please share with me your thoughts about suggested design. Any > > > improvements, missed corner cases or other drawbacks are really > > > appreciated. > > > > > > Thanks, > > > Sergey. > > > > > >
