Trunk has now been replaced. The only changes I'll make now are documentation, release notes, build scripts, license header checks and key signing.
I could use some help as i am time poor. Presently I'm marking bugs on Jira as resolved, then I'll generate the release notes. This release has been more thoroughly tested than any previous River release After 3 is released, we could really utilise your experience Patricia, especially with JERI's ConnectionManager and Multiplexer, it uses complex shared and nested locks, supports 128 concurrent connections (shared endpoints) over one Socket or Channel. If I can get contention with 2 cpu's, it's only going to get worse in a real world situation. This contention would only affect nodes that share multiple remote objects between them. I suspect Gregg's use case will have multiple connections between node pairs and hit this contention, I also suspect that Greg is likely to only have 1 or 2 connections between nodes. Once two nodes have more than 128 connections (connections are directly proportional to the number of remote objects, server or client shared between two nodes) another multiplexer will be created, and so on, multiplexers sync on the ConnectionManager's monitor. I have a Sun T5240 (128 way 64GB Ram), with DilOS (an Illumos based distro with Debian package management). Soon, I should have an high speed ipv6 connection courtesy of the NBN. When I do I'll set you up with a remote login for testing. I'll delay further discussion of security until after 3 is released. The changes I propose will have no bearing (won't be in their call stack) on those who aren't concerned about security. I'll be gratefull for an opportunity to present my security code, perhaps doing so may even dispell some fears. Regards, Peter. Sent from my Samsung device. Include original message ---- Original message ---- From: Patricia Shanahan <p...@acm.org> Sent: 05/12/2015 01:37:10 am To: dev@river.apache.org Subject: Re: Trunk merge and thread pools If you have a real world workload that shows contention, we could make serious progress on performance improvement - after 3.0 ships. I am not even disagreeing with changes that are only shown to make the tests more effective - after 3.0 ships. I am unsure about whether Peter is tilting at windmills or showing the optimum future direction for River with his security ideas. I would be happy to discuss the topic - after 3.0 ships. River 2.2.2, was released November 18, 2013, over two years ago. There is already a lot of good stuff in 3.0 that should be available to users. I have a feeling at this point that we will still be discussing what should be in 3.0 this time next year. In order to get 3.0 out, I believe we need to freeze it. That means two types of changes only until it ships - changes related to organizing the release and fixes for deal-killing regression bugs. If I had the right skills and knowledge to finish up the release I would do it. I don't. Ironically, I do know about multiprocessor performance - I was performance architect for the Sun E10000 and SunFire 15k. Given a suitable benchmark environment, I would love to work on contention - after 3.0 ships. Patricia On 12/4/2015 6:19 AM, Gregg Wonderly wrote: > With a handful of clients, you can ignore contention. My > applications have 20s of threads per client making very frequent > calls through the service and this means that 10ms delays evolve into > seconds of delay fairly quickly. > > I believe that if you can measure the contention with tooling, on > your desktop, it is a viable goal to reduce it or eliminate it. > > It's like system time vs user time optimizations of old. Now we are > contending for processor cores instead of the processor, locked in > the kernel, unable to dispatch more network traffic where it is > always convenient to bury latency. > > Gregg > > Sent from my iPhone > > On Dec 4, 2015, at 9:57 AM, Greg Trasuk <tras...@stratuscom.com> > wrote: > >>> On Dec 4, 2015, at 1:16 AM, Peter <j...@zeus.net.au> wrote: >>> >>> Since ObjectInputStream is a big hotspot, for testing purposes, >>> I merged these changes into my local version of River, my >>> validating ObjectInputStream outperforms the standard java ois >>> >>> Then TaskManager, used by the test became a problem, with tasks >>> in contention up to 30% of the time. >>> >>> Next I replaced TaskManager with an ExecutorService (River 3, >>> only uses TaskManager in tests now, it's no longer used by >>> release code), but there was still contention although not quite >>> as bad. >>> >>> Then I notice that tasks in the test call Thread.yield(), which >>> tends to thrash, so I replaced it with a short sleep of 100ms. >>> >>> Now monitor state was a maximum of 5%, much better. >>> >>> After these changes, the hotspot consuming 27% cpu was JERI's >>> ConnectionManager.connect, followed by Class.getDeclaredMethod >>> at 15.5%, Socket.accept 14.4% and Class.newInstance at 10.8%. >> >> >> First - performance optimization: Unless you’re testing with >> real-life workloads, in real-ife-like network environments, you’re >> wasting your time. In the real world, clients discover services >> pretty rarely, and real-world architects always make sure that >> communications time is small compared to processing time. In the >> real world, remote call latency is controlled by network bandwidth >> and the speed of light. Running in the integration test >> environment, you’re seeing processor loads, not network loads. >> There isn’t any need for this kind of micro-optimization. All >> you’re doing is delaying shipping, no matter how wonderful you keep >> telling us it is. >> >> >>> My validating ois, originating from apache harmony, was modified >>> to use explicit constructors during deserialization. This >>> addressed finalizer attacks, final field immutability and input >>> stream validation and the ois itself places a limit on downloaded >>> bytes by controlling
