Perhaps we can learn something from UDT's algorithms. Rsync is another algorithm that could be worth looking into.
With SDM, if knowledge of state is lost (an event is missed) a lookup is performed to re-establish the current state. Although I chose an arbitrary, short wait time, it could be dynamically tuned. It's main intent is to reorder events arriving out of order, in close proximity, when the sender has processed them asynchronously. With the advent of lambda expressions, my personal thoughts; service lookup can be performed much more efficiently than today. Presently, the lookup service has all necessary data, although filtering is performed, it's based on marshalled form, so no logical comparison is possible. As a result, a client is expected to download this data, as well as codebases for all services before final filtering can be performed. Delayed unmarshalling goes a long way to solving the problem. Going further, although not practical with the present jvm implementation (because it serializes the clients enclosing class as well as the lambda), we can implement remote lambda's while remaining compliant with the java language and jvm spec, by using lambda code extracted from java bytecode, capturing and inlining all variables, before serialising it, then dynamically compile using ASM, create a class and object, we can avoid unnecessary serialization of services, entries and codebases. This also makes SDM obsolete. For remote code downloads, given that isolates have never been released, while multiple jvm instances can now share classes and there are ongoing issues with ClassLoader separation, a separate process for containing smart proxy's is appealing. Not only does this prevent local application classes from being visible to smart proxy code, it would allow the smart proxy and application to use their own versions of libraries, module systems etc. A separate process also allows the user to kill a misbehaving smart proxy application, without killing the application that instantiated it. The local application can communicate with the smart proxy via another reflective proxy, which can implement additional administration interfaces to manage the smart proxy's jvm. In addition, if the delegate reflective proxy is reserialized, it will need to be substituted with the marshalled smart proxy. A separate process provides another significant advantage; hot upgrades. Developers with Objects implementing Distributed won't need to concern themselves with serial form compatibility issues, making an upgrade simply a case of saving state, then reloading with new codebases in a new process. With Distributed, you don't even need the original classes present, you can substitute another implementation, using a builder or static factory method. Even a client application could respawn itself with a new process, performing a hot upgrade. The possibilities are there, we can create an enterprise class distributed system infrasture like no other. Cheers, Peter. ----- Original message ----- > Out of order processing always implies a waiting time and timeout events > processing. What is hard, is discovering how long to wait. What hasn’t > been happening by and large, is to have self adjusting wait times that > will allow network latency to be considered and handled effectively. > Additionally, there is the whole issue of UDP (unacknowledged, > non-streaming data flow) verses TCP (streaming, guaranteed ordering) as > a transport. > > There are several places in this time I’m my career where I’ve actually > needed and wanted to write a PID based networking retry and flow > management library. The idea being that one can learn about intervals > of time that typically are in processing loops across networking stacks > and adjust software behaviors to deal with anomalous behaviors and > optimize for patterns of success or failure. TCP’s > > Syn-> > <-SynACK > SynACK-> > > Pattern is an algorithm which attempts to make sure that traffic is > flowing in both directions and that both ends of a TCP socket connection > agree on the state of the socket. However, the synack timeout on the > connection initiator causes “resources” to be reserved for a period of > time. In the beginning, the TCP connection timeout was on the order of > minutes, because “modems” had to dial the connection in many cases. In > this day and age, the timeout can be more practically set to be on the > order of a few seconds because of the speed of networks and the “always > connected” state. > > But, imagine what could happen if the timeout is set too small. > Suddenly, networking stops functioning to certain places where the > timeout is not long enough. This timeout could be adjusted in length by > first noting how fast connections occur. It could be lengthened if the > SynACK comes back after the timeout, but that could only happen if we > still retained knowledge of the last attempted connections that failed > due to timeout, or else malicious SynACK attacks could create problems. > > So, typically, behavior based tuning of networking algorithms has not > been done because there is always a “hard” case that seems to create > enough complication to keep things from being “helpful” overall. > > I think that we really need to think more about how to make > notifications more stateless so that there is an opportunity to not have > to consider history. > > Gregg Wonderly > > On Apr 19, 2014, at 4:44 AM, Peter <j...@zeus.net.au> wrote: > > > Thanks Gregg, > > > > For now, with SDM I'm using a PriorityBlockingQueue for buffering > > events. Incoming threads originating from remote calls, place an > > event into the buffer, sleep briefly, then take an event from the > > buffer and continue their execution path through SDM. I did consider > > using a handoff thread, but there's no contention in any tests, so I > > kept it simple. While in the buffer, the events are re ordered by the > > Comparator. > > > > When SDM receives events out of order it performs a lookup network > > call, reordering helps to avoid that. > > > > Refactoring SDM to fix race conditions and sync bugs has taken some > > months, but it's sorted now. > > > > The latest issue now appears to be related to building on Jenkins, > > creating jar files and Processes using AppClassLoader resulting in > > ClassNotFoundExceptions. > > > > The cause of CNFE dosn't appear related to River code, but I'm > > investigating whether there are any issues with the build process, it > > doesn't appear to be a classdep or asm problem, as all not found > > classes are accounted for, it could be a classpath issue. > > > > If you profile Jini / River code now, the hotspots are all java > > sockets, no River code is a hotspot, ClassLoading is magnitudes of > > order faster and there is no contention whatsoever. > > > > Cheers, > > > > Peter. > > > > ----- Original message ----- > > > The simple programming mechanism I use for unordered but inclusive > > > events is a map or set. I use a map for what is expected and a map > > > for what has happened. I use a thread responding to notifications > > > to fill in the results data and then either call out or notify > > > another thread of the new results. It’s that final code activity > > > that checks for “do I have everything I need to do more work?” It > > > will then react when that moment occurs, retry, re-dispatch or > > > whatever the appropriate action is. That way, everything is > > > separated and still involves testable behaviors. > > > > > > Gregg Wonderly > > > > > > On Apr 17, 2014, at 7:56 PM, Peter <j...@zeus.net.au> wrote: > > > > > > > Thanks Greg, I agree, remote events have an event id and sequence > > > > number, so it's very easy for clients to order them if necessary. > > > > > > > > I think for the test I'll create a simple comparator that orders > > > > the events at the client. > > > > > > > > The test only needs to ensure that all expected events are received > > > > and provide sufficient information allowing them to be correctly > > > > ordered. > > > > > > > > Regards, > > > > > > > > Peter. > > > > > > > > ----- Original message ----- > > > > > > > > > > Hi Peter: > > > > > > > > > > You should probably create a JIRA enhancement ticket to track > > > > > discussion if you’re picturing adding some kind of > > > > > order-guaranteeing comparator to the API. But I don’t think > > > > > you really need to do that, because the usage would be so > > > > > dependent on the client’s architecture that it probably isn’t > > > > > sensible to put it in the API. > > > > > > > > > > On the actual question, I’d suggest that Reggie should make no > > > > > guarantees on the order of event delivery (as per the event > > > > > spec). That being the case, imposing some kind of order is a > > > > > client problem, not Reggie’s. I would suggest modifying the > > > > > test simply to ensure that all the expected events have been > > > > > received in the required time, regardless of the order. > > > > > Perhaps also add some clarification to the service registrar > > > > > spec. > > > > > > > > > > Cheers, > > > > > > > > > > Greg Trasuk. > > > > > > > > > > On Apr 17, 2014, at 7:30 AM, Peter <j...@zeus.net.au> wrote: > > > > > > > > > > > > > > > > > > > > > > > From: Peter <j...@zeus.net.au> > > > > > > Subject: RemoteEvent specification - proposal > > > > > > Date: April 17, 2014 at 7:28:13 AM EDT > > > > > > To: d...@apache.river.org > > > > > > Reply-To: Peter <j...@zeus.net.au> > > > > > > > > > > > > > > > > > > The Jini Remote Event specification clearly states that remote > > > > > > events may arrive out of order, yet some lookup tck tests in > > > > > > the qa test suite require events to arrive in order. > > > > > > > > > > > > Presently I have an Executor in Reggie, used specifically for > > > > > > sending event notifications, however it is single threaded, to > > > > > > ensure events arrive in an order identical to client > > > > > > registration, to avoid qa test failures. > > > > > > > > > > > > I propose creating a comparator clients can use to order > > > > > > events as they arrive. This will allow qa tests, when > > > > > > utilising this comparator, to pass when Reggie is configured > > > > > > to use a multi threaded event notifier executor. This would > > > > > > increase Reggie's scalability for event notifications. > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > Regards, > > > > > > > > > > > > Peter. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
