Sergey Korotkov created IGNITE-16424:
----------------------------------------
Summary: Generate test data in several threads in
DataGenerationApplication
Key: IGNITE-16424
URL: https://issues.apache.org/jira/browse/IGNITE-16424
Project: Ignite
Issue Type: Improvement
Reporter: Sergey Korotkov
Assignee: Sergey Korotkov
DataGenerationApplication app is used to fill ignite cluster with test data.
Now It accepts as parameters:
* number of caches to create
* number of entries to put to each cache (interval of integer keys in fact)
* size of each cache entry
* number of backup partitions for caches created
Currently application creates and fills caches one by one in one thread. For
huge caches it is a very time-consuming operation. On the other hand it's known
that the parallell load in several threads via the Ignite Streamer works fine
and can speed up the process significantly.
It's also known that such parallell operations are very heap-memory intensive.
Special attention should be paid to this issue.
---
So, the tasks looks like:
# Modify the DataGenerationApplication as it would load data in several
threads. It should accept new integer parameter: _*threads*_ (1 by default) and
use this number of threads to load data trying to spread work evenly between
them.
# Try to figure out some heuristic of the dependence of the required heap
memory on the number of threads, caches and data size. It may require to
implement some tool to get actual heap usage metrics from
DataGenerationApplication after test run.
# Implement this heuristic in python (to start DataGenerationApplication with
the appropriate heap memory JVM options)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
