Two comments only; 1. Object creation is not the only benchmark, is it? In "real" applications, we often see that the performance is dying in the hands of other subsystems, such as serialization, I/O, recursive/complex algorithms and very seldom the management of objects and method invocations. I don't recall the exact path that a method takes, but I am pretty sure it isn't 50 times that of a single call. And that is also hard to measure, since it is difficult to know when hotspot will and will not skew benchmark loops by all kind of eliminations...
2. Architecture; Last year I designed an financial application platform, which had uniform nodes. All nodes did all things, and scaled horizontally without any human effort. The key to get that to work was dynamic and high performance queues and distributed maps. All ValueComposites easily convert to/from JSON, and with this fact all work was pushed around in queues as Message with a "type" and a "value", where the type was "deserialized" to a class, which was input to ValueBuilderFactory.fromJSON( class, json ) (IIRC). For commands, there was a single void method for execute() and for events there were other expected behaviors. Now, we also did a "push command to where data resides" by strictly following Aggregate pattern. I.e. all commands operated on an Aggregate, and all parts of the Aggregate resided (storage and cache wise) on a single node (with back nodes). So, when a command arrives from the outside, 1. Instantiate the Command object with the Aggregate ID and all needed arguments. 2. Pass the command.toJSON() to the queue connecting to the node where the Aggregate "lives". 3. Done. On the receiving side; 1. Read queue. 2. Take "type" field and do Class.forName(type). 3. Do Command c = vbf.fromJSON(class, value) 4. c.execute(); 5. Done. Now, we also required that no hardware failures would corrupt the message flows or loose any data, so each of those sequences was wrapped with a JVM-level transaction, which also meant that any events emitted as a result of the command would not be forwarded until the transaction completed. This made the window, where too many messages would be sent in case of hardware failure, very short and considered acceptable (duplicates were not a big concern). A tricky bit was some services which had to be "single instance only" (or 2, 3) in the cluster; but that can also be solved with distributed queues and maps, effectively by a distributed AtomicLong that each node would acquire as a CountDownLatch. And of course, since Qi4j has very well-defined semantics on services, entities and values, it wasn't hard to wire in this advanced behavior with Qi4j's hooks, so end of the day, the guy writing the app didn't think much about all these going on under the covers. The application ended up being a bit different to what you are used to, but the power of the architecture is enormous. Development is light-weight. Deployment is a breeze, and we can simply add more instances in production if we run out of juice. The bottleneck is a couple of seldom used locks, and scaled well up to 50 boxes in tests and 15 boxes in actual production, and processing about 5000msg/sec during market hours. The reason for the high number of boxes is not due to Qi4j being slow, but that some very very slow analytics happened on every 1000 msg or so, taking 5-20 seconds of raw CPU. We used Hazelcast on that, which is well suited for this, but I think many other choices are available. Cheers Niclas On Tue, Oct 30, 2012 at 5:39 PM, jj <[email protected]> wrote: > Hi Niclas, > > thank you for your email. Well, my SQL EntityStore + SQL Indexing > was solved my the sql-support project added by Paul. The configuration > is indeed a bit tricky and as you and Paul assumed the problem > has something to do with the in-memory config entitystore (or the > corresponding > visibility). Thank you guys again for the prompt resolution. > > 2012/10/29 Niclas Hedhman [via Qi4j-dev] > <[email protected]>: >> On Tue, Oct 30, 2012 at 12:33 AM, Jiri Jetmar <[hidden email]> wrote: >>> Thank you for your advice. Not sure what you mean with "cut" away - >> >> What I meant was; You sent a mail that you get a StackOverflow. I >> looked at the code and went; "Well, where is everything?". I don't >> have it in front of me, so I can't recall the details... Maybe >> tomorrow morning I can comment better. >> >>> I also agree with your proposal of the assembly structure with some >>> small/open >>> questions like should be the Context Objects also composites ? There >>> are a number >>> of benefits to do so (injections of structures/uof), but also there >>> are some disadvange like e.g. >>> performance issues. >> >> Well, if you choose to use objects for some things, that is fine, BUT >> you can not expect that any of the Qi4j artifacts will then work. >> Since Qi4j owns the call stack for Composites and "Qi4j Objects", >> there are a lot of things that are tied together under the covers. >> >> To give you an understanding, let's try an example; >> >> public class MyMixin implements My { >> public void doSomething() { >> : >> } >> >> public void doAnother() { >> this.doSomething(); >> } >> } >> >> Now, IF doSomething() is a Composite Type method, then all the >> Concerns and Constraints of that method will be invoked, even when the >> method call is happening from within the same Java object instance. >> >> Another example; >> Let's say that you have an Application Layer and a Domain Layer. In >> the Domain Layer you have a module private Entity (My1) and a >> application visible Entity (My2). >> Now, the Application Layer creates the UnitOfWork, but the Application >> Layer doesn't have visibility to My1. So, what happens under the >> covers is that; As the methods are invoked and Visibility scope >> changes, the UnitOfWork is changing with it, without loosing track of >> Visibility scope. >> >> These are example of how involved Qi4j Runtime is in the invocation of >> methods on Qi4j artifacts. All parts of the Runtime expects certain >> behavior, and creating a UnitOfWork from a stored away >> UnitOfWorkFactory is illegal and don't call us when you loose the data >> or application doesn't work. >> >> Perhaps we need to spend time documenting the "border" between regular >> Java and Qi4j in more precise manner. >> > > Ok, I see the point and also the danger by not using the composites objects > accordingly. Yes, some additional documentation is always really useful - > at least for the tricky parts. > >> >>> As I understood from the performance test the >>> overhead using >>> composites versus "naked" objects is about 50:1. But sure, if I can >>> benefit from clear >>> structures in the code, clear object ownership etc. I;m ready to add >>> additional boxed >>> to compensate the performance "penalties" using composites everywhere >>> (if there are >>> really performance issues, what is not clear yet). >> >> Which performance test? >> > > org.qi4j.test.performance.runtime.composite.CompositeCreationPerformanceTest > > and the corresponding output on my machine : > > Minimum Java Object Creation Time:89 nanoseconds per object > Minimum Java Object Creation Time:1 nanoseconds per object > Minimum Composite Creation Time:2518 nanoseconds per composite > Minimum Composite Creation Time:2000 nanoseconds per composite > Minimum Qi4j Object Creation Time:1694 nanoseconds per object > Minimum Qi4j Object Creation Time:1389 nanoseconds per object > Minimum Value Creation Time:12153 nanoseconds per composite > Minimum Value Creation Time:10290 nanoseconds per composite > Minimum Composite (builder) Creation Time:1749 nanoseconds per composite > Minimum Composite (builder) Creation Time:1626 nanoseconds per composite > Minimum Value (builder) Creation Time:9217 nanoseconds per composite > Minimum Value (builder) Creation Time:8279 nanoseconds per composite > Transient: 50x > TransientBuilder: 37x > Value: 249x > ValueBuilder: 194x > Object: 34x > > So I intepret the output that e.g. for Transient Composite Objects the > overhead (versus plain "new") is about 50x. > > The test method uses > > TransientBuilder<AnyComposite> builder = > module.newTransientBuilder( AnyComposite.class ); > builder.newInstance(); > > So I guess I intepret it correctly ? > > > > >>> I;m also thinking on a kind of vertical layering as we have more >>> Application here like : >>> >>> Accounting >>> Billing >>> ProductManagement >> >> There are many ways to structure that; >> * Separate JVMs >> * Separate Qi4j Runtime instances in the same JVM >> * Separate Layer "columns" in a single Qi4j Runtime. >> * Different modules in Domain layer. >> >> Once you analyze correlated applications, you often find that they are >> much more complicated than "simple silos", and layers above the domain >> layer will in fact be modeled differently than such silos. >> > > Yes, that can be indeed that case, but we need split or do a kind of > partitioning to have clear ownership of i.) services and the domain (sub) > model. But it might be required to have different partitioning views, > according > the stage (dev, test, prod, ..) > > >> >>> And I would like to "assembly" them togerher (in terms of a Qi4j >>> Application) during development >>> but later in production they would be seperated - each in his own >>> node, therefore separed (J)VMs >>> and "synchronized" using the even-sourcing pattern. >> >> That is one possibility. But don't shard the application because you >> think you need to run them on different hosts due to performance. > > Well, I would of course prefer to have a large shared-memory box that > can handle all the load (not consider here any hardware failure). > But at some point u have to add several boxed to handle the clients requests. > The question is now how/what is the partitioning unit. > > That> sounds like a bad architectural choice. I have not studied this domain >> well to give useful architecture choices, but I tend to use network >> queues (e.g. ZeroMQ) a lot and have all nodes in the cluster >> identical. >> > > Yes, to have identical nodes in the cluster makes the staging process > indeed easier. > Actually we are thinking about to have dedicated (in terms of domain > (sub)model) > components that are offering services only for that particular (sub)domain. > Like > > Accounting offers DoubleEntryBookingService, AccountAdministrationService etc. > > and > > PartnerManagement offers PartnerAdministrationService, etc.. > > What you need then is a kind of WorkflowServices that are using the > above Services. Something > like > > CreateNewPartnerWorkflowService(someContract) { > > Partner partner = > PartnerAdministrationService.createNewPartner(someContract.name()..) > > Account accout = AccountAdministrationService.createNewAccount(partner).. > > .. > } > > > Sure, one can deploy all the components to one node and this is > exactly what we want to do during > development. During development we would like to test the whole domain > withhin seconds, without > any heavyweight infrastructure. I can painfully remember when I has to > develop for a weblogic EJB server > with rountrip cycles of minutes even for a very small change. Never again ! > :-) > > I also agree with you that we first need to figure out the > "performance" patterns our application produces and > then decide what would be best in terms of scalability. I guess this > would be always a compromise as there are several dimentions > to consider like complexity of deployment vs. performance patterns vs. > availalbe boxes vs. current load vs. costs... > > Thank you a lot for the very valuable input ! > > Cheers, > Jiri > >> >> Cheers >> -- >> Niclas Hedhman, Software Developer >> 河南南路555弄15号1901室。 >> http://www.qi4j.org - New Energy for Java >> >> I live here; http://tinyurl.com/3xugrbk >> I work here; http://tinyurl.com/6a2pl4j >> I relax here; http://tinyurl.com/2cgsug >> >> _______________________________________________ >> qi4j-dev mailing list >> [hidden email] >> http://lists.ops4j.org/mailman/listinfo/qi4j-dev >> >> >> ________________________________ >> If you reply to this email, your message will be added to the discussion >> below: >> http://qi4j-dev.23929.n6.nabble.com/SQL-Indexing-SQL-EntityStore-tp7140p7152.html >> To unsubscribe from SQL Indexing + SQL EntityStore, click here. >> NAML > > > > > -- > View this message in context: > http://qi4j-dev.23929.n6.nabble.com/SQL-Indexing-SQL-EntityStore-tp7140p7153.html > Sent from the Qi4j-dev mailing list archive at Nabble.com. > _______________________________________________ > qi4j-dev mailing list > [email protected] > http://lists.ops4j.org/mailman/listinfo/qi4j-dev -- Niclas Hedhman, Software Developer 河南南路555弄15号1901室。 http://www.qi4j.org - New Energy for Java I live here; http://tinyurl.com/3xugrbk I work here; http://tinyurl.com/6a2pl4j I relax here; http://tinyurl.com/2cgsug _______________________________________________ qi4j-dev mailing list [email protected] http://lists.ops4j.org/mailman/listinfo/qi4j-dev
