Re: Setting performance expectations
It means ehCache is better than ignite performance atleast in LOCAL mode :) I fully expect ehCache, Guava Cache, etc to be faster than Ignite since Ignite has more to do to communicate with the grid. But the scale of the difference is the concern, Thus far my experiments do not show it's appreciably faster than reading from the database. -- Daryl Stultz Principal Software Developer _ OpenTempo, Inc http://www.opentempo.com<http://www.opentempo.com/> mailto:daryl.stu...@opentempo.com<mailto:daryl.stu...@opentempo.com>
Ignite Web Agent
Hello, I'm trying to run the Web Console locally using these instructions: https://apacheignite-tools.readme.io/docs/getting-started#section-ignite-web-agent Apache Ignite Tools Documentation<https://apacheignite-tools.readme.io/docs/getting-started#section-ignite-web-agent> apacheignite-tools.readme.io Apache Ignite a memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads, delivering in-memory speeds at petabyte scale. I got the docker container running easy enough, although it looks just like a "demo". Maybe that's because I don't have the agent running yet, which brings me to my problem. I can't find any place to download the Ignite Web Agent, or clone a repo or anything. Where do I find this? Thanks. -- Daryl Stultz Principal Software Developer _ OpenTempo, Inc http://www.opentempo.com<http://www.opentempo.com/> mailto:daryl.stu...@opentempo.com<mailto:daryl.stu...@opentempo.com>
Re: Setting performance expectations
I've discovered that "partitioned" is the default Cache Mode. I set it to "local". Things run faster now. Still not as fast as expected. 2018-09-21 20:54:41.733 DEBUG - c.o.i.IgniteSpec : load 428765 rows into local map in 976 ms 2018-09-21 20:54:43.562 DEBUG - c.o.i.IgniteSpec : putAll map to cache in 1829 ms 2018-09-21 20:54:44.031 DEBUG - c.o.i.IgniteSpec : get 428765 elements one at a time in 469 ms 2018-09-21 20:54:44.313 DEBUG - c.o.i.IgniteSpec : get all keys at once in 282 ms Here are metrics for the same test run with Ehcache. How can I get these same numbers from Ignite? 2018-09-21 21:28:37.273 DEBUG - c.o.i.EhcacheSpec : load 428765 rows into local map in 988 ms 2018-09-21 21:28:38.299 DEBUG - c.o.i.EhcacheSpec : putAll map to cache in 1025 ms 2018-09-21 21:28:38.326 DEBUG - c.o.i.EhcacheSpec : get 428765 elements one at a time in 27 ms 2018-09-21 21:28:38.536 DEBUG - c.o.i.EhcacheSpec : get all keys at once in 210 ms /Daryl
Re: Setting performance expectations
is my understanding correct, that you first select 400k records from DB, DB
is remote and this takes 700ms, then you select all these 400k records from
Ignite and now it takes 500ms, but in this case ignite is a local embedded
node?
All of this runs on my laptop, no network involved. MacBook Pro 2018,
PostgreSQL 9.6. VM flags "-XX:+PrintGC -Xmx5000m -Xms4000m"
Your suggestion to print GC logging led me to increase min and max memory and
free up RAM by closing other apps. Things sped up quite a bit. Does this mean
Ignite was serializing and writing to disk? I didn't have onHeapCacheEnabled
for this run, does that mean it's serializing but to off-heap memory, if memory
is available? After adding setOnheapCacheEnabled(true) the metrics didn't
change much. So here, database to map is 953 ms, and get from cache one at a
time is 723ms, not much faster. Getting all in one batch is twice as fast as
loading from the DB, but I still expect reading from Ignite to be far faster
than the DB.
Here is the output for the below example:
2018-09-20 20:14:23.079 DEBUG - c.o.i.IgniteSpec : load 428765 rows into local
map in 953 ms
[GC (Allocation Failure) 1047513K->282437K(4275200K), 0.1436511 secs]
2018-09-20 20:14:25.363 DEBUG - c.o.i.IgniteSpec : putAll map to cache in 2284
ms
2018-09-20 20:14:26.086 DEBUG - c.o.i.IgniteSpec : get 428765 elements one at a
time in 723 ms
2018-09-20 20:14:26.575 DEBUG - c.o.i.IgniteSpec : get all keys at once in 489
ms
[20:14:26] Ignite node stopped OK [uptime=00:00:04.770]
2018-09-20 20:14:26.805 DEBUG - c.o.w.d.Profiler : [[ cleanDatabase >>> END: 1;
Total:1 ]]
2018-09-20 20:14:26.805 DEBUG - c.o.u.s.EbeanExtension : END TransactionEmpty
test suite.
2018-09-20 20:14:26.820 DEBUG - i.e.s.l.ShutdownManager : Shutting down
2018-09-20 20:14:26.820 DEBUG - i.e.s.c.DefaultServer : Shutting down
EbeanServer unittest-ws
[GC (System.gc()) 992502K->265050K(4616704K), 0.0797061 secs]
[Full GC (System.gc()) 265050K->171852K(4616704K), 0.3171326 secs]
Here's the code. It's Kotlin. (I did do a very basic get/put test of Kotlin vs
Java, though no DB involved, performance was comparable.)
data class ScheduledAssignmentDto(
val id: Int,
val userId: Int,
val roleId: Int,
val date: Date
)
class IgniteSpec : Specification({
val log = Log(IgniteSpec::class.java)
val config = IgniteConfiguration()
config.setClientMode(false)
val ignite = Ignition.start(config)
val cacheCfg = CacheConfiguration()
cacheCfg.name = "ScheduledAssignmentDto"
val cache = ignite.getOrCreateCache(cacheCfg)
val sql = """select scheduledAssignmentId as id, userId, roleId,
ScheduleDays.scheduleDate as date
from ScheduledAssignments
join ScheduleDays on ScheduleDays.scheduleDayId =
ScheduledAssignments.scheduleDayId
order by scheduledAssignmentId
""".trimMargin()
var rowCount = 1
val keys = mutableSetOf()
val map = mutableMapOf()
val loadTime = measureTimeMillis {
TenantResourceSupplier.getEbeanServer().createSqlQuery(sql)
.findEachRow { resultSet: ResultSet, rowNumber: Int ->
val id = resultSet.getInt("id")
val userId = resultSet.getInt("userId")
val roleId = resultSet.getInt("roleId")
val date = Date(resultSet.getDate("date").time)
val dto = ScheduledAssignmentDto(id, userId, roleId, date)
map.put(id, dto)
rowCount = rowNumber
keys += id
}
}
log.debug("load $rowCount rows into local map in $loadTime ms")
val cacheMapTime = measureTimeMillis {
cache.putAll(map)
}
log.debug("putAll map to cache in $cacheMapTime ms")
val readTime = measureTimeMillis {
keys.forEach {
cache.get(it)
}
}
log.debug("get $rowCount elements one at a time in $readTime ms")
val readTime2 = measureTimeMillis {
cache.getAll(keys)
}
log.debug("get all keys at once in $readTime2 ms")
ignite.close()
})
Thanks for the help.
/Daryl
Setting performance expectations
Hello, I am trying out Ignite for the first time. I have some interesting performance metrics that are unexpected. I'm looking to see if my understanding of what Ignite is meant to do is correct. I have 400K records from a database that I am loading into a cache with the primary key mapped to a DTO of a few properties from the record. For my real world scenario I need to perform a lot of calculations in memory of data. The bottleneck by far is the time it takes to load data from the database. (So I want to do repeated calculations with cached data.) If I read all 400K rows from the database and simply discard the DTO it takes about 700 ms. Reading all rows into a HashMap is about 1100 ms. If I then putAll the map to the cache it's 3100 ms. If I then getAll keys from the cache it's 550 ms. The cache in this case is "embedded", i.e., just the one JVM I'm running the test on (clientMode false). I have a lot of other numbers but these are best case. Going to a server node on the same machine takes an order of magnitude longer if getting and putting in a loop rather than batch. I didn't test remote caches. I'm not looking for the best way to warm/pre-load a cache, I'm just looking to see how the basics work. Reading a row at a time from the database is 30 times faster than reading a key at a time from the cache. As such Ignite will not help me with my problem. Do my numbers seem normal? Is this a use case you would expect Ignite to help with? I ran IgnitePutBenchmark and the report says 124K ops per second average. I'm certainly not seeing that in my test case. Thanks. -- Daryl Stultz Principal Software Developer _ OpenTempo, Inc http://www.opentempo.com<http://www.opentempo.com/> mailto:daryl.stu...@opentempo.com<mailto:daryl.stu...@opentempo.com>
