timoninmaxim commented on code in PR #11057:
URL: https://github.com/apache/ignite/pull/11057#discussion_r1409846870
##########
modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java:
##########
@@ -80,7 +87,7 @@ public DumpReader(DumpReaderConfiguration cfg, IgniteLogger
log) {
cnsmr.start();
- try {
+ try (StatsLogger statsLogger = new StatsLogger()) {
Review Comment:
Abbreviation should be used.
##########
modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java:
##########
@@ -109,17 +116,26 @@ public DumpReader(DumpReaderConfiguration cfg,
IgniteLogger log) {
AtomicBoolean skip = new AtomicBoolean(false);
- Map<Integer, Set<Integer>> groups = cfg.skipCopies() ? new
HashMap<>() : null;
+ Map<Integer, Set<Integer>> grpToParts = cfg.skipCopies() ? new
HashMap<>() : null;
+
+ int partsCnt = grpToNodes.entrySet().stream().mapToInt(e ->
+ (int)e.getValue().stream()
+ .flatMap(node -> dump.partitions(node,
e.getKey()).stream())
+ .filter(part -> grpToParts == null ||
grpToParts.computeIfAbsent(e.getKey(), x -> new HashSet<>()).add(part))
+ .count())
+ .sum();
- if (groups != null)
- grpToNodes.keySet().forEach(grpId -> groups.put(grpId, new
HashSet<>()));
+ AtomicInteger partsProcessed = new AtomicInteger(0);
+
+ if (grpToParts != null)
+ grpToParts.clear();
Review Comment:
Looks weird that you fill this collection in 124L, clear here, and fill
again in 138L.
##########
modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java:
##########
@@ -109,17 +116,26 @@ public DumpReader(DumpReaderConfiguration cfg,
IgniteLogger log) {
AtomicBoolean skip = new AtomicBoolean(false);
- Map<Integer, Set<Integer>> groups = cfg.skipCopies() ? new
HashMap<>() : null;
Review Comment:
Let's keep the previous name of the var to reduce amount of changes
##########
modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java:
##########
@@ -135,13 +151,51 @@ public DumpReader(DumpReaderConfiguration cfg,
IgniteLogger log) {
return;
}
- try (DumpedPartitionIterator iter =
dump.iterator(node, grp, part)) {
+ try (DumpedPartitionIterator iter = new
DumpedPartitionIterator() {
+ /** */
+ final DumpedPartitionIterator delegate =
dump.iterator(node, grp, part);
+
+ /** */
+ final AtomicBoolean
consumerProcessingEntry = new AtomicBoolean(false);
Review Comment:
Can we count on `next()` only?
##########
modules/core/src/main/java/org/apache/ignite/dump/DumpReader.java:
##########
@@ -135,13 +151,51 @@ public DumpReader(DumpReaderConfiguration cfg,
IgniteLogger log) {
return;
}
- try (DumpedPartitionIterator iter =
dump.iterator(node, grp, part)) {
+ try (DumpedPartitionIterator iter = new
DumpedPartitionIterator() {
+ /** */
+ final DumpedPartitionIterator delegate =
dump.iterator(node, grp, part);
+
+ /** */
+ final AtomicBoolean
consumerProcessingEntry = new AtomicBoolean(false);
+
+ /** {@inheritDoc } */
+ @Override public boolean hasNext() {
+ if
(consumerProcessingEntry.compareAndSet(true, false))
+ statsLogger.recordProcessed();
+
+ return delegate.hasNext();
+ }
+
+ /** {@inheritDoc } */
+ @Override public DumpEntry next() {
+ if
(consumerProcessingEntry.compareAndSet(true, false)) {
+ // Consumer didn't execute
hasNext()
+ statsLogger.recordProcessed();
+ }
+
+ DumpEntry next = delegate.next();
+
+ consumerProcessingEntry.set(true);
+
+ return next;
+ }
+
+ /** {@inheritDoc } */
+ @Override public void close() throws
Exception {
+ delegate.close();
+ }
+ }) {
if (log.isDebugEnabled()) {
log.debug("Consuming partition [node="
+ node + ", grp=" + grp +
", part=" + part + ']');
}
cnsmr.onPartition(grp, part, iter);
+
+ int partNo =
partsProcessed.incrementAndGet();
+
+ if (log.isInfoEnabled())
+ log.info("Consumed partitions " +
partNo + " of " + partsCnt);
Review Comment:
I see the pretty common log message in 188L. I suppose we need to log the
partition counter at the same rate as entry counter. If user needs to log every
partition he/she could use debug level of logging. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
