is my understanding correct, that you first select 400k records from DB, DB
is remote and this takes 700ms, then you select all these 400k records from
Ignite and now it takes 500ms, but in this case ignite is a local embedded
node?
All of this runs on my laptop, no network involved. MacBook Pro 2018,
PostgreSQL 9.6. VM flags "-XX:+PrintGC -Xmx5000m -Xms4000m"
Your suggestion to print GC logging led me to increase min and max memory and
free up RAM by closing other apps. Things sped up quite a bit. Does this mean
Ignite was serializing and writing to disk? I didn't have onHeapCacheEnabled
for this run, does that mean it's serializing but to off-heap memory, if memory
is available? After adding setOnheapCacheEnabled(true) the metrics didn't
change much. So here, database to map is 953 ms, and get from cache one at a
time is 723ms, not much faster. Getting all in one batch is twice as fast as
loading from the DB, but I still expect reading from Ignite to be far faster
than the DB.
Here is the output for the below example:
2018-09-20 20:14:23.079 DEBUG - c.o.i.IgniteSpec : load 428765 rows into local
map in 953 ms
[GC (Allocation Failure) 1047513K->282437K(4275200K), 0.1436511 secs]
2018-09-20 20:14:25.363 DEBUG - c.o.i.IgniteSpec : putAll map to cache in 2284
ms
2018-09-20 20:14:26.086 DEBUG - c.o.i.IgniteSpec : get 428765 elements one at a
time in 723 ms
2018-09-20 20:14:26.575 DEBUG - c.o.i.IgniteSpec : get all keys at once in 489
ms
[20:14:26] Ignite node stopped OK [uptime=00:00:04.770]
2018-09-20 20:14:26.805 DEBUG - c.o.w.d.Profiler : [[ cleanDatabase >>> END: 1;
Total:1 ]]
2018-09-20 20:14:26.805 DEBUG - c.o.u.s.EbeanExtension : END TransactionEmpty
test suite.
2018-09-20 20:14:26.820 DEBUG - i.e.s.l.ShutdownManager : Shutting down
2018-09-20 20:14:26.820 DEBUG - i.e.s.c.DefaultServer : Shutting down
EbeanServer unittest-ws
[GC (System.gc()) 992502K->265050K(4616704K), 0.0797061 secs]
[Full GC (System.gc()) 265050K->171852K(4616704K), 0.3171326 secs]
Here's the code. It's Kotlin. (I did do a very basic get/put test of Kotlin vs
Java, though no DB involved, performance was comparable.)
data class ScheduledAssignmentDto(
val id: Int,
val userId: Int,
val roleId: Int,
val date: Date
)
class IgniteSpec : Specification({
val log = Log(IgniteSpec::class.java)
val config = IgniteConfiguration()
config.setClientMode(false)
val ignite = Ignition.start(config)
val cacheCfg = CacheConfiguration<Int, ScheduledAssignmentDto>()
cacheCfg.name = "ScheduledAssignmentDto"
val cache = ignite.getOrCreateCache<Int, ScheduledAssignmentDto>(cacheCfg)
val sql = """select scheduledAssignmentId as id, userId, roleId,
ScheduleDays.scheduleDate as date
from ScheduledAssignments
join ScheduleDays on ScheduleDays.scheduleDayId =
ScheduledAssignments.scheduleDayId
order by scheduledAssignmentId
""".trimMargin()
var rowCount = 1
val keys = mutableSetOf<Int>()
val map = mutableMapOf<Int, ScheduledAssignmentDto>()
val loadTime = measureTimeMillis {
TenantResourceSupplier.getEbeanServer().createSqlQuery(sql)
.findEachRow { resultSet: ResultSet, rowNumber: Int ->
val id = resultSet.getInt("id")
val userId = resultSet.getInt("userId")
val roleId = resultSet.getInt("roleId")
val date = Date(resultSet.getDate("date").time)
val dto = ScheduledAssignmentDto(id, userId, roleId, date)
map.put(id, dto)
rowCount = rowNumber
keys += id
}
}
log.debug("load $rowCount rows into local map in $loadTime ms")
val cacheMapTime = measureTimeMillis {
cache.putAll(map)
}
log.debug("putAll map to cache in $cacheMapTime ms")
val readTime = measureTimeMillis {
keys.forEach {
cache.get(it)
}
}
log.debug("get $rowCount elements one at a time in $readTime ms")
val readTime2 = measureTimeMillis {
cache.getAll(keys)
}
log.debug("get all keys at once in $readTime2 ms")
ignite.close()
})
Thanks for the help.
/Daryl
