[
https://issues.apache.org/jira/browse/IGNITE-22047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-22047:
---------------------------------------
Description:
This happens in some runs of ItSqlLogicTest:
[2024-04-15T13:34:27,352][ERROR][%sqllogic1%low-watermark-updater-0][LowWatermarkImpl]
Failed to update low watermark, will schedule again: HybridTimestamp
[physical=2024-04-15 13:34:26:299 +0000, logical=0,
composite=112275493112971264]
java.util.concurrent.CompletionException:
org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-1
TraceId:95c41ebc-944f-4618-9639-00a706332244 Operation has been cancelled (node
is stopping).
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
~[?:?]
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
~[?:?]
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1739)
~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: org.apache.ignite.internal.lang.IgniteInternalException: Operation
has been cancelled (node is stopping).
at
org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:828)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.busy(AbstractPageMemoryTableStorage.java:218)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.destroyIndex(AbstractPageMemoryTableStorage.java:183)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.engine.ThreadAssertingMvTableStorage.destroyIndex(ThreadAssertingMvTableStorage.java:96)
~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.TableImpl.unregisterIndex(TableImpl.java:292)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.index.IndexManager.destroyIndex(IndexManager.java:238)
~[ignite-index-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$5(IndexManager.java:227)
~[ignite-index-3.0.0-SNAPSHOT.jar:?]
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at
org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$6(IndexManager.java:227)
~[ignite-index-3.0.0-SNAPSHOT.jar:?]
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
~[?:?]
... 3 more
Caused by: org.apache.ignite.internal.lang.NodeStoppingException: Operation has
been cancelled (node is stopping).
at
org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:828)
~[ignite-core-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.busy(AbstractPageMemoryTableStorage.java:218)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.destroyIndex(AbstractPageMemoryTableStorage.java:183)
~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.storage.engine.ThreadAssertingMvTableStorage.destroyIndex(ThreadAssertingMvTableStorage.java:96)
~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.table.TableImpl.unregisterIndex(TableImpl.java:292)
~[ignite-table-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.index.IndexManager.destroyIndex(IndexManager.java:238)
~[ignite-index-3.0.0-SNAPSHOT.jar:?]
at
org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$5(IndexManager.java:227)
~[ignite-index-3.0.0-SNAPSHOT.jar:?]
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
at
org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$6(IndexManager.java:227)
~[ignite-index-3.0.0-SNAPSHOT.jar:?]
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
~[?:?]
... 3 more
The problem seems to be that when, upon an attempt to raise LWM, we try to
destroy an index (that was dropped earlier individualy, not as a part of its
table drop), it turns out that the table has already also been dropped (as DROP
TABLE happened after DROP INDEX). Index destruction gets bounced because the
table storage is already closed.
This causes the LWM raise attempt to fail forever.
We probably should not fail if the destroyIndex() call fails to take the busy
lock; instead, we should just ignore the failure; this is because the table
storage is either under destruction (so the table will be destroyed together
with its indexes), or the node is being stopped (and so the index will be
destroyed upon restart).
> Exception is thrown on index destruction if the table storage is already
> closed
> -------------------------------------------------------------------------------
>
> Key: IGNITE-22047
> URL: https://issues.apache.org/jira/browse/IGNITE-22047
> Project: Ignite
> Issue Type: Bug
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> This happens in some runs of ItSqlLogicTest:
>
> [2024-04-15T13:34:27,352][ERROR][%sqllogic1%low-watermark-updater-0][LowWatermarkImpl]
> Failed to update low watermark, will schedule again: HybridTimestamp
> [physical=2024-04-15 13:34:26:299 +0000, logical=0,
> composite=112275493112971264]
> java.util.concurrent.CompletionException:
> org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-1
> TraceId:95c41ebc-944f-4618-9639-00a706332244 Operation has been cancelled
> (node is stopping).
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
> ~[?:?]
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1739)
> ~[?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [?:?]
> at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: org.apache.ignite.internal.lang.IgniteInternalException: Operation
> has been cancelled (node is stopping).
> at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:828)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.busy(AbstractPageMemoryTableStorage.java:218)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.destroyIndex(AbstractPageMemoryTableStorage.java:183)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.engine.ThreadAssertingMvTableStorage.destroyIndex(ThreadAssertingMvTableStorage.java:96)
> ~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.table.TableImpl.unregisterIndex(TableImpl.java:292)
> ~[ignite-table-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.index.IndexManager.destroyIndex(IndexManager.java:238)
> ~[ignite-index-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$5(IndexManager.java:227)
> ~[ignite-index-3.0.0-SNAPSHOT.jar:?]
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
> at
> org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$6(IndexManager.java:227)
> ~[ignite-index-3.0.0-SNAPSHOT.jar:?]
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
> ~[?:?]
> ... 3 more
> Caused by: org.apache.ignite.internal.lang.NodeStoppingException: Operation
> has been cancelled (node is stopping).
> at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLock(IgniteUtils.java:828)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.busy(AbstractPageMemoryTableStorage.java:218)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.pagememory.AbstractPageMemoryTableStorage.destroyIndex(AbstractPageMemoryTableStorage.java:183)
> ~[ignite-storage-page-memory-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.storage.engine.ThreadAssertingMvTableStorage.destroyIndex(ThreadAssertingMvTableStorage.java:96)
> ~[ignite-storage-api-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.table.TableImpl.unregisterIndex(TableImpl.java:292)
> ~[ignite-table-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.index.IndexManager.destroyIndex(IndexManager.java:238)
> ~[ignite-index-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$5(IndexManager.java:227)
> ~[ignite-index-3.0.0-SNAPSHOT.jar:?]
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
> at
> org.apache.ignite.internal.index.IndexManager.lambda$onLwmChanged$6(IndexManager.java:227)
> ~[ignite-index-3.0.0-SNAPSHOT.jar:?]
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
> ~[?:?]
> ... 3 more
>
> The problem seems to be that when, upon an attempt to raise LWM, we try to
> destroy an index (that was dropped earlier individualy, not as a part of its
> table drop), it turns out that the table has already also been dropped (as
> DROP TABLE happened after DROP INDEX). Index destruction gets bounced because
> the table storage is already closed.
> This causes the LWM raise attempt to fail forever.
> We probably should not fail if the destroyIndex() call fails to take the busy
> lock; instead, we should just ignore the failure; this is because the table
> storage is either under destruction (so the table will be destroyed together
> with its indexes), or the node is being stopped (and so the index will be
> destroyed upon restart).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
