bamaer commented on code in PR #9527:
URL: https://github.com/apache/hop/pull/9527#discussion_r4596520535
##########
engine/src/main/java/org/apache/hop/pipeline/SingleThreadedPipelineExecutor.java:
##########
@@ -369,6 +382,21 @@ public boolean oneIteration() throws HopException {
return false;
}
}
+
+ // All input arrived and was read, but the transform can still
have work: a reader
+ // going through the file a row named, a mapping, the rows a Table
output still holds.
+ // Like a transform without input, it is called until it says it
is done. Otherwise it
+ // never sees the end of its input, and the single-threaded
engine, which runs one
+ // iteration, disposes it with that work undone.
+ //
+ if (!transformDone &&
isAllInputRead(combi.transform.getInputRowSets())) {
+ while (!transformDone && !pipeline.isStopped()) {
+ transformDone = !combi.transform.processRow();
+ if (combi.transform.getErrors() > 0) {
+ return false;
+ }
+ }
+ }
Review Comment:
**Blocker:** a transform that finishes in this loop still gets
`batchComplete()` further down, in the same iteration. Group by, Memory group
by, Unique rows and Row denormaliser write their last rows in the end-of-input
`processRow()`, then write them again in `batchComplete()`.
Data grid (a,1 a,2 b,3 b,4) → Group by `grp`, sum `val`, on the
single-threaded engine:
- main: `a 3`, `b 7`
- PR: `a 3`, `b 7`, `b 11` (count 3)
Memory group by gives every group twice. Unique rows gives the last row
twice.
Suggested fix: don't call `batchComplete()` for a transform this loop just
finished.
```java
finishedAtEndOfInput = transformDone; // after the while loop
...
if (!finishedAtEndOfInput) {
combi.transform.batchComplete();
}
```
A pipeline in `integration-tests/single_threaded` would catch this: that
Data grid into Group by, Memory group by and Unique rows, expecting 2 rows from
each.
##########
engine/src/main/java/org/apache/hop/pipeline/SingleThreadedPipelineExecutor.java:
##########
@@ -299,6 +299,19 @@ public boolean init() {
*
* @return true if more iterations can be performed. False if this is not
the case.
*/
+ /**
+ * Whether every input row set is done and has no rows left: the transform
has read all its input.
+ * A transform removes a row set it read to the end, so no row sets left
means the same.
+ */
+ private static boolean isAllInputRead(List<IRowSet> rowSets) {
+ for (IRowSet rowSet : rowSets) {
+ if (!rowSet.isDone() || rowSet.size() > 0) {
Review Comment:
**Blocker:** in the batch callers (Kafka consumer, Event Hubs listener), a
row set can be marked done between batches, because `SortRows.batchComplete()`
calls `setOutputDone()` after every batch. After batch 1, the transform after
Sort rows counts as having read all its input. It finishes and is skipped for
every later batch.
Injector → Sort rows → Dummy, 4 batches of 3 rows, with `oneIteration()`
after each batch as `KafkaConsumerInput` does. Rows out of Dummy:
- main: 3 / 6 / 9 / 12
- PR: 3 / 3 / 3 / 3
Suggested fix: also require the transform that writes into the row set to be
done.
```java
private boolean isAllInputRead(List<IRowSet> rowSets) {
for (IRowSet rowSet : rowSets) {
if (!rowSet.isDone() || rowSet.size() > 0 || !isOriginDone(rowSet)) {
return false;
}
}
return true;
}
private boolean isOriginDone(IRowSet rowSet) {
for (int i = 0; i < transforms.size(); i++) {
TransformMetaDataCombi c = transforms.get(i);
if (c.transformName.equals(rowSet.getOriginTransformName())
&& c.copy == rowSet.getOriginTransformCopy()) {
return done[i];
}
}
return true; // a RowProducer: done once finished() was called
}
```
With both changes the batch case gives 12/12 rows and the single-threaded
engine gives 2 correct rows per branch. A unit test that runs Injector → Sort
rows → Dummy through a few `oneIteration()` calls would cover this without
Kafka.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]