bamaer commented on code in PR #9527:
URL: https://github.com/apache/hop/pull/9527#discussion_r4596520535


##########
engine/src/main/java/org/apache/hop/pipeline/SingleThreadedPipelineExecutor.java:
##########
@@ -369,6 +382,21 @@ public boolean oneIteration() throws HopException {
                 return false;
               }
             }
+
+            // All input arrived and was read, but the transform can still 
have work: a reader
+            // going through the file a row named, a mapping, the rows a Table 
output still holds.
+            // Like a transform without input, it is called until it says it 
is done. Otherwise it
+            // never sees the end of its input, and the single-threaded 
engine, which runs one
+            // iteration, disposes it with that work undone.
+            //
+            if (!transformDone && 
isAllInputRead(combi.transform.getInputRowSets())) {
+              while (!transformDone && !pipeline.isStopped()) {
+                transformDone = !combi.transform.processRow();
+                if (combi.transform.getErrors() > 0) {
+                  return false;
+                }
+              }
+            }

Review Comment:
   **Blocker:** a transform that finishes in this loop still gets 
`batchComplete()` further down, in the same iteration. Group by, Memory group 
by, Unique rows and Row denormaliser write their last rows in the end-of-input 
`processRow()`, then write them again in `batchComplete()`.
   
   Data grid (a,1 a,2 b,3 b,4) → Group by `grp`, sum `val`, on the 
single-threaded engine:
   - main: `a 3`, `b 7`
   - PR: `a 3`, `b 7`, `b 11` (count 3)
   
   Memory group by gives every group twice. Unique rows gives the last row 
twice.
   
   Suggested fix: don't call `batchComplete()` for a transform this loop just 
finished.
   ```java
                 finishedAtEndOfInput = transformDone; // after the while loop
   ...
               if (!finishedAtEndOfInput) {
                 combi.transform.batchComplete();
               }
   ```
   A pipeline in `integration-tests/single_threaded` would catch this: that 
Data grid into Group by, Memory group by and Unique rows, expecting 2 rows from 
each.



##########
engine/src/main/java/org/apache/hop/pipeline/SingleThreadedPipelineExecutor.java:
##########
@@ -299,6 +299,19 @@ public boolean init() {
    *
    * @return true if more iterations can be performed. False if this is not 
the case.
    */
+  /**
+   * Whether every input row set is done and has no rows left: the transform 
has read all its input.
+   * A transform removes a row set it read to the end, so no row sets left 
means the same.
+   */
+  private static boolean isAllInputRead(List<IRowSet> rowSets) {
+    for (IRowSet rowSet : rowSets) {
+      if (!rowSet.isDone() || rowSet.size() > 0) {

Review Comment:
   **Blocker:** in the batch callers (Kafka consumer, Event Hubs listener), a 
row set can be marked done between batches, because `SortRows.batchComplete()` 
calls `setOutputDone()` after every batch. After batch 1, the transform after 
Sort rows counts as having read all its input. It finishes and is skipped for 
every later batch.
   
   Injector → Sort rows → Dummy, 4 batches of 3 rows, with `oneIteration()` 
after each batch as `KafkaConsumerInput` does. Rows out of Dummy:
   - main: 3 / 6 / 9 / 12
   - PR: 3 / 3 / 3 / 3
   
   Suggested fix: also require the transform that writes into the row set to be 
done.
   ```java
     private boolean isAllInputRead(List<IRowSet> rowSets) {
       for (IRowSet rowSet : rowSets) {
         if (!rowSet.isDone() || rowSet.size() > 0 || !isOriginDone(rowSet)) {
           return false;
         }
       }
       return true;
     }
   
     private boolean isOriginDone(IRowSet rowSet) {
       for (int i = 0; i < transforms.size(); i++) {
         TransformMetaDataCombi c = transforms.get(i);
         if (c.transformName.equals(rowSet.getOriginTransformName())
             && c.copy == rowSet.getOriginTransformCopy()) {
           return done[i];
         }
       }
       return true; // a RowProducer: done once finished() was called
     }
   ```
   With both changes the batch case gives 12/12 rows and the single-threaded 
engine gives 2 correct rows per branch. A unit test that runs Injector → Sort 
rows → Dummy through a few `oneIteration()` calls would cover this without 
Kafka.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to