[
https://issues.apache.org/jira/browse/IGNITE-29281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko updated IGNITE-29281:
-------------------------------------
Description:
A race between query failure handling and {{QueryStartResponse}} processing can
leave a failed query registered on a remote node.
After a DELETE fails, {{CalciteQueryProcessor.processQuery()}} unregisters the
client’s {{RootQuery}} while it is still {{CLOSING}} and waiting for a fragment
start response. The response handler then cannot find the query, so cleanup
does not complete and {{QueryCloseMessage}} is never sent.
Besides test cleanup failures, this may retain query objects and execution
resources, increase memory usage after repeated failures, and leave stale
entries in query monitoring.
The race was observed without artificial delays while running in a container
with two CPUs available on relatively low-powered hardware. I have not
explicitly observed it on TeamCity. The attached synthetic reproducer delays
{{QueryStartResponse}} until a DELETE fails, then releases it and checks
cleanup. It fails before the fix and passes afterward.
See [^QueryCleanupRaceIntegrationTest.java]
was:
A race between query failure handling and {{QueryStartResponse}} processing can
leave a failed query registered on a remote node.
After a DELETE fails, {{CalciteQueryProcessor.processQuery()}} unregisters the
client’s {{RootQuery}} while it is still {{CLOSING}} and waiting for a fragment
start response. The response handler then cannot find the query, so cleanup
does not complete and {{QueryCloseMessage}} is never sent.
Besides test cleanup failures, this may retain query objects and execution
resources, increase memory usage after repeated failures, and leave stale
entries in query monitoring.
The race was observed without artificial delays while running in a container
with two CPUs available on relatively low-powered hardware. I have not
explicitly observed it on TeamCity. The attached synthetic reproducer delays
{{QueryStartResponse}} until a DELETE fails, then releases it and checks
cleanup. It fails before the fix and passes afterward.
> SQL Calcite: Query cleanup race
> -------------------------------
>
> Key: IGNITE-29281
> URL: https://issues.apache.org/jira/browse/IGNITE-29281
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 2.18, 2.19
> Reporter: Kirill Tkalenko
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: calcite, ignite-2, sql
> Attachments: QueryCleanupRaceIntegrationTest.java
>
>
> A race between query failure handling and {{QueryStartResponse}} processing
> can leave a failed query registered on a remote node.
> After a DELETE fails, {{CalciteQueryProcessor.processQuery()}} unregisters
> the client’s {{RootQuery}} while it is still {{CLOSING}} and waiting for a
> fragment start response. The response handler then cannot find the query, so
> cleanup does not complete and {{QueryCloseMessage}} is never sent.
> Besides test cleanup failures, this may retain query objects and execution
> resources, increase memory usage after repeated failures, and leave stale
> entries in query monitoring.
> The race was observed without artificial delays while running in a container
> with two CPUs available on relatively low-powered hardware. I have not
> explicitly observed it on TeamCity. The attached synthetic reproducer delays
> {{QueryStartResponse}} until a DELETE fails, then releases it and checks
> cleanup. It fails before the fix and passes afterward.
> See [^QueryCleanupRaceIntegrationTest.java]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
