I am removing about 100 thousands records, using a data streamer as follows:
> try (
> QueryCursor
> <
> List<?
> >
>> cursor = cache.query(new SqlFieldsQuery("SELECT _key FROM MyClass WHERE x
>> = ?").setArgs(y));
> IgniteDataStreamer
> <
> MyKey, MyClass
> >
> streamer = ignite.dataStreamer(cache.getName());
> ) {
> streamer.allowOverwrite(true);
> for (List<?> fieldsResult : cursor) {
> MyKey key = (MyKey) fieldsResult.get(0);
> streamer.removeData(key);
> }
> }
I noticed this seems to be taking a while (~ 1 min) so I tried a different
approach just to see how it does.
> try (
> QueryCursor
> <
> List<?
> >
>> cursor = cache.query(new SqlFieldsQuery("SELECT _key FROM MyClass WHERE x
>> = ?").setArgs(y));
> ) {
> Set
> <MyKey>
> keys = new HashSet<>();
>
> for (List<?> fieldsResult : cursor) {
> keys.add((MyKey) fieldsResult.get(0));
> }
>
> cache.clearAll(keys);
> }
Using clearAll instead took about 1s. This was surprising since I thought
data streamer is supposed to be faster? Could someone explain the big
difference in speed here? Am I missing something?
I tried it with both 1 and 2 nodes and the results were about the same.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/DataStreamer-performance-with-removing-data-tp1917.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
