Hi Denis,
Some progress had happened and I have some material to share with the
community. I think it will be interesting to newbies. It is about loading
big tables from rdbms and creating cache entries based on table info. This
approach was tested in production and showed good timing being paired with
MSSQL, tables from tens to hundreds million rows.
The loading jar process:
* starts Ignite client node;
* creates user POJO according to business logic;
* converts POJOs to BinaryObjects;
* uses affinity function and creates separate key-value HashMap for every
cache partition;
* uses ComputeTaskAdaper/ComputeJobAdaper to place hashMaps on
corresponding data node.
I would like to publish some tutorial, say on GridGain website in english
and russian version on habr.com.
WDYT?
чт, 12 мар. 2020 г. в 08:25, :
> Hello Denis,
>
> That is possible, my writing activities should be continued. The only
> question is to get my local project to production, there is no sense in
> writing another model example. So I hope there will be a progress in the
> nearest future
>
> Vladimir
>
> 2:25, 12 марта 2020 г., Denis Magda :
>
> Hello Vladimir,
>
> Just to clarify, are you suggesting to create a tutorial for data loading
> scenarios when data resides in an external database?
>
> -
> Denis
>
>
> On Tue, Mar 10, 2020 at 11:41 PM wrote:
>
> Andrei, Evgenii, thanks for answer.
>
> Aa far as I see, there is no ready to use tutorial. I managed to do
> multi-threaded cache load procedure, out-of-the-box loadCache method is
> extremely slow.
>
> I spent about a month studying write-through topics, and finally got the
> same as "capacity planning" says: 0.8Gb mssql table on disk expands to
> 2.3Gb, size in ram is 2.875 times bigger.
>
> Is it beneficial to use BinaryObject instead of user pojo? If yes, how to
> create BinaryObject without pojo definition and deserialize it back to pojo?
> It would be great to have kind of advanced github example like this
>
> https://github.com/dmagda/MicroServicesExample
>
> It helped a lot in understanding. Current documentation links do not help
> to build a real solution, they are mostly like a reference, with no option
> to compile and debug
>
> Vladimir
>
> 2:51, 11 марта 2020 г., Evgenii Zhuravlev :
>
> When you're saying that the result was poor, do you mean that data
> preloading took too much time, or it's just about get operations?
>
> Evgenii
>
> вт, 10 мар. 2020 г. в 03:29, aealexsandrov :
>
> Hi,
>
> You can read the documentation articles:
>
> https://apacheignite.readme.io/docs/3rd-party-store
>
> In case if you are going to load the cache from 3-rd party store (RDBMS)
> then the default implementation of CacheJdbcPojoStore can take a lot of
> time
> for loading the data because it used JDBC connection inside (not pull of
> these connections).
>
> Probably you should implement your own version of CacheStore that will read
> data from RDBMS in several threads, e.g using the JDBC connection pull
> there. Sources are open for you, so you can copy the existed implementation
> and modify it:
>
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java
>
> Otherwise, you can do the initial data loading using some streaming tools:
>
> 1)Spark integration with Ignite -
> https://apacheignite-fs.readme.io/docs/ignite-data-frame
> 2)Kafka integration with Ignite -
> https://apacheignite-mix.readme.io/docs/kafka-streamer
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>
> --
> Отправлено из мобильного приложения Яндекс.Почты
>
>
>
> --
> Отправлено из мобильного приложения Яндекс.Почты
