[
https://issues.apache.org/jira/browse/UNOMI-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Huber updated UNOMI-959:
------------------------------
Description:
h3. Summary for anyone triaging this
One of our automated tests — which checks that adding a customer profile to a
marketing list works correctly through the GraphQL API — occasionally fails for
no clear reason, both on our CI servers and on developers' own machines. It
doesn't fail because the feature is broken; when you retry, it usually passes.
This is a "flaky test": something intermittently goes wrong either in the test
itself or in how the system responds under load, and right now we don't know
which. It's a nuisance for anyone running the build (a red CI run with no real
bug behind it wastes time and erodes trust in the test suite), and it's worth
root-causing before it masks a real regression later.
h3. Technical details
{{org.apache.unomi.itests.graphql.GraphQLListIT.testCRUD}} fails with:
{code:java}
java.lang.AssertionError: Failed waiting for profile in list query (last value:
...)
at org.apache.unomi.itests.BaseIT.keepTrying(BaseIT.java:812)
at
org.apache.unomi.itests.graphql.GraphQLListIT.testCRUD(GraphQLListIT.java:83)
{code}
The test adds a profile to a list via the {{addProfileToList}} GraphQL
mutation, then polls the {{findLists}} query (with a nested {{active}}
profile-membership check) for up to ~60 seconds waiting for the profile to show
up as a list member. On a failing run, karaf.log shows the underlying
OpenSearch query returning a structural {{match-none}} result on *every single*
polling attempt for the entire retry window:
{code:java}
WARN TypeResolutionServiceImpl - Condition has no type ID for query builder
WARN ConditionOSQueryBuilderDispatcher - Condition type is null for condition
typeID=null, returning match-none query
{code}
That is, the query itself appears to be structurally incapable of matching
anything for the whole window — not a normal indexing/refresh delay that should
resolve on retry. Reproduced on both CI and local runs; timing/system-load
appears to be a factor (failures line up with periods of elevated CPU/memory
pressure per collected metrics).
*Impact:* Wastes CI/developer time chasing a false-positive failure; risks the
team learning to ignore red runs from this test, which would hide a real
regression if one occurred.
*Steps to reproduce:* Run the full IT suite ({{itests}} module, {{AllITs}})
under load; failure rate is inconsistent (not every run).
*Acceptance Criteria:*
* Root cause of the intermittent {{match-none}} query result identified with
reproducible evidence
* {{GraphQLListIT.testCRUD}} passes reliably across repeated local/CI runs
under load
* Regression coverage added for whatever the confirmed mechanism turns out to be
was:
h3. Summary for anyone triaging this
One of our automated tests — which checks that adding a customer profile to a
marketing list works correctly through the GraphQL API — occasionally fails for
no clear reason, both on our CI servers and on developers' own machines. It
doesn't fail because the feature is broken; when you retry, it usually passes.
This is a "flaky test": something intermittently goes wrong either in the test
itself or in how the system responds under load, and right now we don't know
which. It's a nuisance for anyone running the build (a red CI run with no real
bug behind it wastes time and erodes trust in the test suite), and it's worth
root-causing before it masks a real regression later.
h3. Technical details:
`org.apache.unomi.itests.graphql.GraphQLListIT.testCRUD` fails with:
{code:java}
java.lang.AssertionError: Failed waiting for profile in list query (last value:
...)
at org.apache.unomi.itests.BaseIT.keepTrying(BaseIT.java:812)
at
org.apache.unomi.itests.graphql.GraphQLListIT.testCRUD(GraphQLListIT.java:83)
{code}
The test adds a profile to a list via the {{addProfileToList}} GraphQL
mutation, then polls the {{findLists}} query (with a nested {{active}}
profile-membership check) for up to ~60 seconds waiting for the profile to show
up as a list member. On a failing run, karaf.log shows the underlying
OpenSearch query returning a structural {{match-none}} result on *every single*
polling attempt for the entire retry window:
{code:java}
WARN TypeResolutionServiceImpl - Condition has no type ID for query builder
WARN ConditionOSQueryBuilderDispatcher - Condition type is null for condition
typeID=null, returning match-none query
{code}
That is, the query itself appears to be structurally incapable of matching
anything for the whole window — not a normal indexing/refresh delay that should
resolve on retry. Reproduced on both CI and local runs; timing/system-load
appears to be a factor (failures line up with periods of elevated CPU/memory
pressure per collected metrics).
*{*}Impact:{*}* Wastes CI/developer time chasing a false-positive failure;
risks the team learning to ignore red runs from this test, which would hide a
real regression if one occurred.
*{*}Steps to reproduce:{*}* Run the full IT suite (`itests` module, `AllITs`)
under load; failure rate is inconsistent (not every run).
*{*}Acceptance Criteria:{*}*
- [ ] Root cause of the intermittent `match-none` query result identified with
reproducible evidence
- [ ] `GraphQLListIT.testCRUD` passes reliably across repeated local/CI runs
under load
- [ ] Regression coverage added for whatever the confirmed mechanism turns out
to be
> Integration test `GraphQLListIT.testCRUD` fails intermittently under load
> -------------------------------------------------------------------------
>
> Key: UNOMI-959
> URL: https://issues.apache.org/jira/browse/UNOMI-959
> Project: Apache Unomi
> Issue Type: Sub-task
> Components: unomi(-core)
> Affects Versions: unomi-3.1.0
> Reporter: Serge Huber
> Priority: Major
> Fix For: unomi-3.1.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> h3. Summary for anyone triaging this
> One of our automated tests — which checks that adding a customer profile to a
> marketing list works correctly through the GraphQL API — occasionally fails
> for no clear reason, both on our CI servers and on developers' own machines.
> It doesn't fail because the feature is broken; when you retry, it usually
> passes. This is a "flaky test": something intermittently goes wrong either in
> the test itself or in how the system responds under load, and right now we
> don't know which. It's a nuisance for anyone running the build (a red CI run
> with no real bug behind it wastes time and erodes trust in the test suite),
> and it's worth root-causing before it masks a real regression later.
> h3. Technical details
> {{org.apache.unomi.itests.graphql.GraphQLListIT.testCRUD}} fails with:
> {code:java}
> java.lang.AssertionError: Failed waiting for profile in list query (last
> value: ...)
> at org.apache.unomi.itests.BaseIT.keepTrying(BaseIT.java:812)
> at
> org.apache.unomi.itests.graphql.GraphQLListIT.testCRUD(GraphQLListIT.java:83)
> {code}
> The test adds a profile to a list via the {{addProfileToList}} GraphQL
> mutation, then polls the {{findLists}} query (with a nested {{active}}
> profile-membership check) for up to ~60 seconds waiting for the profile to
> show up as a list member. On a failing run, karaf.log shows the underlying
> OpenSearch query returning a structural {{match-none}} result on *every
> single* polling attempt for the entire retry window:
> {code:java}
> WARN TypeResolutionServiceImpl - Condition has no type ID for query builder
> WARN ConditionOSQueryBuilderDispatcher - Condition type is null for condition
> typeID=null, returning match-none query
> {code}
> That is, the query itself appears to be structurally incapable of matching
> anything for the whole window — not a normal indexing/refresh delay that
> should resolve on retry. Reproduced on both CI and local runs;
> timing/system-load appears to be a factor (failures line up with periods of
> elevated CPU/memory pressure per collected metrics).
> *Impact:* Wastes CI/developer time chasing a false-positive failure; risks
> the team learning to ignore red runs from this test, which would hide a real
> regression if one occurred.
> *Steps to reproduce:* Run the full IT suite ({{itests}} module, {{AllITs}})
> under load; failure rate is inconsistent (not every run).
> *Acceptance Criteria:*
> * Root cause of the intermittent {{match-none}} query result identified with
> reproducible evidence
> * {{GraphQLListIT.testCRUD}} passes reliably across repeated local/CI runs
> under load
> * Regression coverage added for whatever the confirmed mechanism turns out to
> be
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
