Re: Is invokeAll() considered a batch operation?
As a side note - can it be so that lambda passed to consume() is executed in 1 thread? I agree with Sergi that string keys should be replaced with ints. Alex G, can you check how we map invoke in atomic cache? Can it be so that processing 27 keys may end up with sending 13 or 14 messages if keys belong to nodes this way - local node, remote node, local, remote... Can we optimize this to send 1 message and then process local set? On Sep 1, 2015 19:31, "Sergi Vladykin" <sergi.vlady...@gmail.com> wrote: > Sorry, just noticed that you already use field values as cache keys. > But simplifying counters to ints instead of HashMaps of HashSets is > probably still makes sense to try. > > Sergi > > 2015-09-01 19:19 GMT+03:00 Sergi Vladykin <sergi.vlady...@gmail.com>: > >> By the way you can also use field id (just number your fields from 0 to >> 27) instead of storing field names in counter keys, it will give some >> serialization speedup and better memory utilization. >> >> Sergi >> >> 2015-09-01 19:10 GMT+03:00 Sergi Vladykin <sergi.vlady...@gmail.com>: >> >>> As I see you have your field names as keys, you can think of them as >>> locks. >>> I mean that your scalability is limited by these 27 locks and you are >>> acquiring them all for each cache update. >>> How many threads do you use for your latency test? >>> I'd suggest to use something like tuple (fieldName, fieldValue) as key >>> and >>> just an integer (number of such keys) as value which must be incremented >>> or set to 1 if none. >>> >>> Sergi >>> >>> >>> 2015-09-01 17:38 GMT+03:00 javadevmtl <java.dev@gmail.com>: >>> >>>> Sorry for the confusion... >>>> >>>> So... >>>> >>>> - Just to be clear running 1.3.0 on 2 nodes with total of 64 cores and >>>> 256GB >>>> of RAM >>>> - Cache is configured as off-heap partitioned no backups with 96GB and >>>> each >>>> node is started as -Xmx4g >>>> - Each app is a web server that receives json request that parses the >>>> json >>>> and inserts the json properties as different keys (27 total). >>>> - Requests are load balanced to both nodes. >>>> >>>> All times quoted below includes full business logic and network time. >>>> The >>>> json payload sent always contains the 27 properties. Only the app is >>>> recompiled to either invokeAll() for 1 key 3 keys, 9 keys etc... >>>> >>>> Also time remains constant from 0 records inserted all the way up to >>>> millions (so at least this is good thing) which also means java is >>>> sufficiently warmed up with millions of calls. >>>> >>>> 01 keys: 2ms >>>> 03 keys: 4ms >>>> 09 keys: 9ms >>>> 18 keys: 16ms >>>> 27 keys: 25ms >>>> >>>> I was maybe hopping slightly better latency... Here is the code maybe >>>> you >>>> have suggestions on improvements? >>>> >>>> http://pastebin.com/pR36DwdG >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-ignite-users.70518.x6.nabble.com/Is-invokeAll-considered-a-batch-operation-tp1220p1238.html >>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>>> >>> >>> >> >
Re: Is invokeAll() considered a batch operation?
Hi Yakov... Each physical server node is 32 cores. When I deploy my vertx.io application I tell it to create 32 instances of consumer.handler Vertx.io is much like node.js reactor pattern. Each request is put into an event loop and each instance of consume.handler will take 1 event from the loop and process it. Vertx underneath handles all the dirty work. So everything that runs inside consume would be as if it was 1 single thread. Though of course ignite will work in it's on thread pool, that doesn't change. As far as the keys goes it's arbitrary data so it's not attached to an identifier. Here are some questions I try to answer Given the person "John Smith" how many unique phone numbers does he have. Given the phone number "555-555-" how many unique persons are associated to it. The other way I was trying to this was using a a data model where each data is serialized, but secondary indexes where proving slow. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Is-invokeAll-considered-a-batch-operation-tp1220p1247.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Is invokeAll() considered a batch operation?
By the way you can also use field id (just number your fields from 0 to 27) instead of storing field names in counter keys, it will give some serialization speedup and better memory utilization. Sergi 2015-09-01 19:10 GMT+03:00 Sergi Vladykin <sergi.vlady...@gmail.com>: > As I see you have your field names as keys, you can think of them as locks. > I mean that your scalability is limited by these 27 locks and you are > acquiring them all for each cache update. > How many threads do you use for your latency test? > I'd suggest to use something like tuple (fieldName, fieldValue) as key and > just an integer (number of such keys) as value which must be incremented > or set to 1 if none. > > Sergi > > > 2015-09-01 17:38 GMT+03:00 javadevmtl <java.dev@gmail.com>: > >> Sorry for the confusion... >> >> So... >> >> - Just to be clear running 1.3.0 on 2 nodes with total of 64 cores and >> 256GB >> of RAM >> - Cache is configured as off-heap partitioned no backups with 96GB and >> each >> node is started as -Xmx4g >> - Each app is a web server that receives json request that parses the json >> and inserts the json properties as different keys (27 total). >> - Requests are load balanced to both nodes. >> >> All times quoted below includes full business logic and network time. The >> json payload sent always contains the 27 properties. Only the app is >> recompiled to either invokeAll() for 1 key 3 keys, 9 keys etc... >> >> Also time remains constant from 0 records inserted all the way up to >> millions (so at least this is good thing) which also means java is >> sufficiently warmed up with millions of calls. >> >> 01 keys: 2ms >> 03 keys: 4ms >> 09 keys: 9ms >> 18 keys: 16ms >> 27 keys: 25ms >> >> I was maybe hopping slightly better latency... Here is the code maybe you >> have suggestions on improvements? >> >> http://pastebin.com/pR36DwdG >> >> >> >> >> >> >> >> >> >> -- >> View this message in context: >> http://apache-ignite-users.70518.x6.nabble.com/Is-invokeAll-considered-a-batch-operation-tp1220p1238.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >
Re: Is invokeAll() considered a batch operation?
Sorry for the confusion... So... - Just to be clear running 1.3.0 on 2 nodes with total of 64 cores and 256GB of RAM - Cache is configured as off-heap partitioned no backups with 96GB and each node is started as -Xmx4g - Each app is a web server that receives json request that parses the json and inserts the json properties as different keys (27 total). - Requests are load balanced to both nodes. All times quoted below includes full business logic and network time. The json payload sent always contains the 27 properties. Only the app is recompiled to either invokeAll() for 1 key 3 keys, 9 keys etc... Also time remains constant from 0 records inserted all the way up to millions (so at least this is good thing) which also means java is sufficiently warmed up with millions of calls. 01 keys: 2ms 03 keys: 4ms 09 keys: 9ms 18 keys: 16ms 27 keys: 25ms I was maybe hopping slightly better latency... Here is the code maybe you have suggestions on improvements? http://pastebin.com/pR36DwdG -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Is-invokeAll-considered-a-batch-operation-tp1220p1238.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Is invokeAll() considered a batch operation?
As I see you have your field names as keys, you can think of them as locks. I mean that your scalability is limited by these 27 locks and you are acquiring them all for each cache update. How many threads do you use for your latency test? I'd suggest to use something like tuple (fieldName, fieldValue) as key and just an integer (number of such keys) as value which must be incremented or set to 1 if none. Sergi 2015-09-01 17:38 GMT+03:00 javadevmtl <java.dev@gmail.com>: > Sorry for the confusion... > > So... > > - Just to be clear running 1.3.0 on 2 nodes with total of 64 cores and > 256GB > of RAM > - Cache is configured as off-heap partitioned no backups with 96GB and each > node is started as -Xmx4g > - Each app is a web server that receives json request that parses the json > and inserts the json properties as different keys (27 total). > - Requests are load balanced to both nodes. > > All times quoted below includes full business logic and network time. The > json payload sent always contains the 27 properties. Only the app is > recompiled to either invokeAll() for 1 key 3 keys, 9 keys etc... > > Also time remains constant from 0 records inserted all the way up to > millions (so at least this is good thing) which also means java is > sufficiently warmed up with millions of calls. > > 01 keys: 2ms > 03 keys: 4ms > 09 keys: 9ms > 18 keys: 16ms > 27 keys: 25ms > > I was maybe hopping slightly better latency... Here is the code maybe you > have suggestions on improvements? > > http://pastebin.com/pR36DwdG > > > > > > > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Is-invokeAll-considered-a-batch-operation-tp1220p1238.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
