Re: Missing records Ignite cache size grows

[Andrey Mashenkov]

Hi,

I've made a repro and it works fine for me.
Please check if I missed smth?

On Wed, Mar 1, 2017 at 5:15 PM, diopek <deha.pe...@gmail.com> wrote:

> Below is our implementation of LoadOnlyCacheStore inputIterator method.
> As we tested many times there is no issue on uniqueness of generated cache
> keys.
> But still after loading high number of records into IgniteCache, records
> count doesn't match, and few 10s of keys were missing out of million
> records. Also if we use the Java cache populated just before serializing
> this cache into IgniteCache, all number of records are matching.
>
> In short, after several days of try-outs and debugging, we narrowed down
> the
> root cause of issue to  LoadOnlyCacheStore base class. So, at this point,
> we
> need Ignite core team support to resolve this bug.
> Thanks
>
>
> Yes, we checked uniqueness of key several times. Below is representative
> snippet of our code, how we generate cache key inside inputIterator method.
>
> @Override
> protected Iterator<Tuple2&lt;Long,ArrayList&lt;MyDTO>>>*
> inputIterator*(Object... args) throws CacheLoaderException {
>         Iterator<Tuple2&lt;Long,ArrayList&lt;MyDTO>>> iterator = null;
>
> ResultSetExtractor<ArrayList&lt;Tuple2&lt;Long,ArrayList&lt;MyDTO>>>>
> extOrRepMapResultSetExtractor = new
> ResultSetExtractor<ArrayList&lt;Tuple2&lt;Long,ArrayList&lt;MyDTO>>>>() {
>                 @Override
>                 public ArrayList<Tuple2&lt;Long,ArrayList&lt;MyDTO>>>
> extractData(ResultSet rs)
>                                 throws SQLException, DataAccessException {
>                         ArrayList<Tuple2&lt;Long,ArrayList&lt;MyDTO>>>
> extOrRepList = new
> ArrayList<Tuple2&lt;Long,ArrayList&lt;MyDTO>>>(startSize);
>                         *final AtomicLong entryCnt = new AtomicLong(0);*
>                         while (rs.next()) {
>                                 extOrRepList.add(new Tuple2<Long,
> ArrayList&lt;MyDTO>>(*entryCnt.incrementAndGet()*,prevDTOList));
>                         }
>                         return extOrRepList;
>                 }
>         };
>
>         jdbcTemplate.setFetchSize(SQL_FETCH_SIZE);
>         ArrayList<Tuple2&lt;Long,ArrayList&lt;MyDTO>>> extOrRepList =
> null;
>         extOrRepList = jdbcTemplate.query(sql,
> extOrRepMapResultSetExtractor);
>
>         if (extOrRepList != null) {
>                 iterator = extOrRepList.iterator();
>         }
>         return iterator;
>
> }
>
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Missing-records-Ignite-cache-size-
> grows-tp10809p10966.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Best regards,
Andrey V. Mashenkov

MissedEntries.java
Description: Binary data