bamaer commented on PR #8953: URL: https://github.com/apache/hop/pull/8953#issuecomment-6209274398
Thanks @mattcasters. The mail, lineage and samples timeout fixes look good. I built the branch and ran the unit tests (engine, neo4j, opensearch, elastic, execution-database: all green) and the ITs with `run-tests-docker.sh`. mail, neo4j (including `error-lineage-8704`), neo4j-bolt, memgraph, falkordb, age and exec-info all pass. **Blockers** 1. **The OpenSearch IT still fails:** the golden set has 3 rows and it received 2, in two Docker runs and three local runs. A logging proxy shows two location instances writing the sub-workflow document. One writes `Running` and then `Finished`. The other loaded the document before it had any state (to attach the actions) and keeps persisting that copy, last of all on close, after the `Finished` write. Before this PR those writes went to a second document. With `_id` now kept, they overwrite the real one. Suggested fix: an instance that loaded a document only to attach children should merge onto a fresh read before it persists (as `keepNewerParentState` does for the link), or not persist the workflow's own state from that copy. 2. **`refresh=wait_for` on every write** (`OpenSearchExecutionInfoLocation.documentPath`): each write now waits for the next refresh, about 1s instead of 6 ms. Twenty child pipelines through a Pipeline Executor take 66–70s instead of 8–9.5s, and a single pipeline takes 7.0s instead of 5.7s. It also doesn't fix the IT (see 1). Suggested fix: drop it. If the link re-read has to see the latest write, use a realtime `GET <index>/_doc/<_id>` or a single `_refresh` before that read. If it helps, the mail, Neo4j and samples changes could go in on their own, with OpenSearch in a follow-up. I'm happy to share the proxy setup and the timing project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
