Hello Vladimir, Sounds interesting, thanks for reaching out. Let me introduce you to @Kseniya Romanova <[email protected]> who can help with the publication process.
- Denis On Sun, Jun 21, 2020 at 10:31 PM Vladimir Tchernyi <[email protected]> wrote: > Hi Denis, > > Some progress had happened and I have some material to share with the > community. I think it will be interesting to newbies. It is about loading > big tables from rdbms and creating cache entries based on table info. This > approach was tested in production and showed good timing being paired with > MSSQL, tables from tens to hundreds million rows. > > The loading jar process: > * starts Ignite client node; > * creates user POJO according to business logic; > * converts POJOs to BinaryObjects; > * uses affinity function and creates separate key-value HashMap for every > cache partition; > * uses ComputeTaskAdaper/ComputeJobAdaper to place hashMaps on > corresponding data node. > > I would like to publish some tutorial, say on GridGain website in english > and russian version on habr.com. > > WDYT? > > чт, 12 мар. 2020 г. в 08:25, <[email protected]>: > >> Hello Denis, >> >> That is possible, my writing activities should be continued. The only >> question is to get my local project to production, there is no sense in >> writing another model example. So I hope there will be a progress in the >> nearest future >> >> Vladimir >> >> 2:25, 12 марта 2020 г., Denis Magda <[email protected]>: >> >> Hello Vladimir, >> >> Just to clarify, are you suggesting to create a tutorial for data loading >> scenarios when data resides in an external database? >> >> - >> Denis >> >> >> On Tue, Mar 10, 2020 at 11:41 PM <[email protected]> wrote: >> >> Andrei, Evgenii, thanks for answer. >> >> Aa far as I see, there is no ready to use tutorial. I managed to do >> multi-threaded cache load procedure, out-of-the-box loadCache method is >> extremely slow. >> >> I spent about a month studying write-through topics, and finally got the >> same as "capacity planning" says: 0.8Gb mssql table on disk expands to >> 2.3Gb, size in ram is 2.875 times bigger. >> >> Is it beneficial to use BinaryObject instead of user pojo? If yes, how to >> create BinaryObject without pojo definition and deserialize it back to pojo? >> It would be great to have kind of advanced github example like this >> >> https://github.com/dmagda/MicroServicesExample >> >> It helped a lot in understanding. Current documentation links do not help >> to build a real solution, they are mostly like a reference, with no option >> to compile and debug >> >> Vladimir >> >> 2:51, 11 марта 2020 г., Evgenii Zhuravlev <[email protected]>: >> >> When you're saying that the result was poor, do you mean that data >> preloading took too much time, or it's just about get operations? >> >> Evgenii >> >> вт, 10 мар. 2020 г. в 03:29, aealexsandrov <[email protected]>: >> >> Hi, >> >> You can read the documentation articles: >> >> https://apacheignite.readme.io/docs/3rd-party-store >> >> In case if you are going to load the cache from 3-rd party store (RDBMS) >> then the default implementation of CacheJdbcPojoStore can take a lot of >> time >> for loading the data because it used JDBC connection inside (not pull of >> these connections). >> >> Probably you should implement your own version of CacheStore that will >> read >> data from RDBMS in several threads, e.g using the JDBC connection pull >> there. Sources are open for you, so you can copy the existed >> implementation >> and modify it: >> >> >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java >> >> Otherwise, you can do the initial data loading using some streaming tools: >> >> 1)Spark integration with Ignite - >> https://apacheignite-fs.readme.io/docs/ignite-data-frame >> 2)Kafka integration with Ignite - >> https://apacheignite-mix.readme.io/docs/kafka-streamer >> >> BR, >> Andrei >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >> >> >> -- >> Отправлено из мобильного приложения Яндекс.Почты >> >> >> >> -- >> Отправлено из мобильного приложения Яндекс.Почты > >
