[
https://issues.apache.org/jira/browse/IGNITE-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anghel Botos updated IGNITE-4521:
---------------------------------
Description:
As suggested in the comments from
https://issues.apache.org/jira/browse/IGNITE-4164, we are using
{{org.apache.ignite.IgniteCache#loadCache}} with a large number of custom
queries to allow for the parallel load of the cache. The custom queries are
created by our code in a similar manner as in
{{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}} when
called with no custom queries. The problem that we face is the fact that the
generation of these queries takes a while (on the order of minutes for our data
set), time which could already be spent loading data into the cache, once the
first queries are available.
The suggestion for improvement is to change the signature of the
{{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}} and
{{org.apache.ignite.IgniteCache#loadCache}} method to use a Producer/Stream for
the custom queries, and the workers in the thread pool doing the actual loading
should just get queries from this stream until it is exhausted/get from the
producer until it returns null (or some other indicator that there are no more
queries to be produced). This would shorten the total time required to load the
cache, as it would make it more similar in behavior to the part of
{{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}}
handling the case without custom queries (where as soon as the boundaries of a
range are retrieved from the initial {{loadCacheSelRangeQry}}, the load of the
range is automatically submitted to the thread pool for execution)
was:As suggested in the comments from When using
org.apache.ignite.IgniteCache#loadCache with
> Allow org.apache.ignite.IgniteCache#loadCache to use also a producer/stream
> of custom queries
> ---------------------------------------------------------------------------------------------
>
> Key: IGNITE-4521
> URL: https://issues.apache.org/jira/browse/IGNITE-4521
> Project: Ignite
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.8
> Reporter: Anghel Botos
>
> As suggested in the comments from
> https://issues.apache.org/jira/browse/IGNITE-4164, we are using
> {{org.apache.ignite.IgniteCache#loadCache}} with a large number of custom
> queries to allow for the parallel load of the cache. The custom queries are
> created by our code in a similar manner as in
> {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}} when
> called with no custom queries. The problem that we face is the fact that the
> generation of these queries takes a while (on the order of minutes for our
> data set), time which could already be spent loading data into the cache,
> once the first queries are available.
> The suggestion for improvement is to change the signature of the
> {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}} and
> {{org.apache.ignite.IgniteCache#loadCache}} method to use a Producer/Stream
> for the custom queries, and the workers in the thread pool doing the actual
> loading should just get queries from this stream until it is exhausted/get
> from the producer until it returns null (or some other indicator that there
> are no more queries to be produced). This would shorten the total time
> required to load the cache, as it would make it more similar in behavior to
> the part of
> {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}}
> handling the case without custom queries (where as soon as the boundaries of
> a range are retrieved from the initial {{loadCacheSelRangeQry}}, the load of
> the range is automatically submitted to the thread pool for execution)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
