weiqingy opened a new issue, #983: URL: https://github.com/apache/flink-agents/issues/983
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description ## Summary `test_long_term_memory_async_execution_in_action` asserts an exact count of long-term memory items, but that count is decided by an LLM and is not stable. The test turns CI red on unrelated changes, including twice on `main`. The assertion is `assert len(items) == 1` in `check_result` (`python/flink_agents/e2e_tests/e2e_tests_integration/long_term_memory_test.py`). It checks how many items mem0 stores after alice's two inputs, "My favorite fruit is watermelon." followed by "My favorite fruit is bananas.". Whether those collapse into one item or stay separate is the extraction model's judgment call. The comment directly above the assertion already concedes this: "LLMs may treat different review comments as updates to the same fact or as distinct facts." ## Evidence The same assertion has produced four different results in three weeks, across three branches: | Date | Branch | Result | |---|---|---| | 2026-07-25 | `fix-react-0.2` | `assert 4 == 1` ([run 30143115864](https://github.com/apache/flink-agents/actions/runs/30143115864)) | | 2026-07-25 | `release-0.2` | `assert 2 == 1` ([run 30144095649](https://github.com/apache/flink-agents/actions/runs/30144095649)) | | 2026-07-25 | `main` | `Py4JJavaError` on execute ([run 30150040643](https://github.com/apache/flink-agents/actions/runs/30150040643)) | | 2026-08-08 | `main` | `assert 0 == 1` ([run 31266039736](https://github.com/apache/flink-agents/actions/runs/31266039736)) | Observed item counts for the same input: 0, 1, 2, and 4. The `--reruns 2` retry added in #717 hides most occurrences rather than removing them. In the run immediately before the 08-08 failure ([31262378878](https://github.com/apache/flink-agents/actions/runs/31262378878)), this test failed its first attempt and passed on retry, and it was the only rerun in that job. On 08-08 all three attempts failed. The 08-08 failure landed on a commit that changed three markdown files under `docs/content` with one line each, so it was clearly unrelated to the change under test. ## Why the count is unstable The fact-extraction model is a remote hosted endpoint (`qwen3.6-plus` over `coding.dashscope.aliyuncs.com`), configured at `long_term_memory_test.py:145-165`. Only the embedding model is local Ollama, despite the resource being named `ollama_qwen3`. Temperature already defaults to 0.1 (`DEFAULT_TEMPERATURE` in `openai_chat_model.py`) and the count still varies, so lowering it further is not a fix. There is no `seed` parameter on the setup. ## Proposed fix Assert on retrievable content rather than an exact count: - Replace `assert len(items) == 1` with a non-empty check plus an assertion that some stored item carries the later fact. - Drop the `created_at < updated_at` assertion, which only holds on the branch where the model consolidated to exactly one item. - Include bob's memory set in the failure message when alice's is empty, so a failure distinguishes a per-key miss from a store-wide one. This removes the `2` and `4` failures, which are legitimate model variance, and keeps a real regression failing: if nothing stores the later fact, the test still fails. ## Deliberately still failing on zero The empty case is kept as a failure rather than tolerated. Zero items means neither add stored anything, which may be a genuine bug in the add path rather than model variance, and it is the only one of the four observed outcomes that carries information. Today an occurrence is not diagnosable. The e2e arm runs with `log_cli_level=OFF` (`tools/ut.sh`), and pytest-rerunfailures reports only the final attempt, so the earlier attempts leave no trace. Adding bob's set to the failure message is the cheap first step: if bob has items and alice does not, the miss is per-key; if both are empty, extraction or the store failed wholesale. Whether the zero case is model variance or a real defect is an open question. The next occurrence with the diagnostic in place should answer it. ## Relation to prior work #716 proposed three mitigations for flaky live-LLM tests. Item 1, per-test retry, shipped in #717 and is what has been absorbing these failures. Item 3, loosening exact-equality assertions on LLM output, was never done. This assertion is exactly that shape. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
