Given that metadata updates will be initiated with DiscoveryCustomEvent, I
see no problems neither with total message ordering (the order will be the
same on all nodes and consistent with discovery events) nor with failover -
the event is marked with @TcpDiscoveryEnsureDelivery, so the event should
be fired even in the case of coordinator failure.
I think handling metadata on clients might be tricky, however fore-fetch
approach is proven to work in the scope of IGNITE-4157.
2017-01-13 12:04 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
> In my understanding the only difficult thing here is how to disallow usage
> of not-yet-confirmed metadata during serialization. As we assume metadata
> updates to be relatively rare, can we always initiate message send from
> coordinator node? In this case we will have total order of all metadata
> updates, what will make conflict resolution trivial.
> Alex, how do you think - will this approach work? I am concerned about
> coordinator failover.
> On Thu, Jan 12, 2017 at 4:09 PM, Sergey Chugunov <
> > Hello Ignite devs,
> > I came up with the following design proposal for IGNITE-4302
> > <https://issues.apache.org/jira/browse/IGNITE-4302>, which is very
> > to described here
> > <http://apache-ignite-developers.2346864.n4.nabble.
> > proposal-td11996.html>
> > proposal for IGNITE-4157 <https://issues.apache.org/
> > jira/browse/IGNITE-4157>,
> > "*Use discovery custom messages instead of marshaller cache*".
> > Protocol is very similar to IGNITE-4157 and based on exchanging two
> > messages: *MetadataProposed*/*MetadataAccepted*.
> > 1. When a node wants to add or update existing metadata, it sends
> > *MetadataProposed* message with the update.
> > 2. Coordinator node checks for any conflicts with existing metadata
> > this typeId. In case of no conflicts coordinator assigns a version
> > number
> > to new metadata and sends Proposed message further.
> > In case of conflict it simply marks this message as IN_CONFLICT and
> > sends it to the original node.
> > 3. All nodes upon receiving Proposed message update local metadata
> > including version number and put it to "pending_acceptance" status.
> > 4. Coordinator acknowledges proposed message with *MetadataAccepted*
> > message, marking it with the version number from proposed message.
> > 5. Each node receiving accepted message checks local version number
> > the one from the message. If they are equal, metadata is considered as
> > accepted by the cluster. 
> > New nodes joining the cluster receive metadata on discovery phase.
> > As discovery messages are delivered to clients asynchronously, there is a
> > possibility that cluster may consider and agree upon update to metadata
> > before client receives initial proposed message.
> > In that case I suggest to attach version number of metadata BinaryObject
> > was serialized with to that BinaryObject so client may check whether it
> > deserialize this object safely or need to request some updated version of
> > metadata.
> >  When a metadata is marked as "pending_acceptance" it should be
> > prohibited to use it to serialize binary objects but it still can be used
> > for deserialization.
> > At the same time from code perspective I didn't find a clear distinction
> > like "this method is used only for serialization and that method is
> > only during deserialization". So for now the only way I see is to block
> > operations for a given typeId until MetadataAccepted message for this
> > typeId is arrived.
> >  If MetadataAccepted message has a version number less than currently
> > pending, it is ignored and all threads waiting for this metadata to be
> > accepted remain blocked.
> > It may be possible for new joining node to receive a MetadataAccepted
> > message with version number greater than local copy, in that case it is
> > fine to apply metadata update and declare metadata accepted.
> > Please share with me your thoughts about suggested design. Any
> > improvements, missed corner cases or other drawbacks are really
> > appreciated.
> > Thanks,
> > Sergey.