[
https://issues.apache.org/jira/browse/IGNITE-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Steshin updated IGNITE-21656:
--------------------------------------
Description:
Ignite may fail to create dump of cache with a node filter.
1) If there are no partitions on some node, we assign the empty set as the
snapshot task result:
{code:java}
if (parts.isEmpty() && !withMetaStorage)
task0 = new GridFinishedFuture<>(Collections.emptySet());
else {
...
}
{code}
Cache dump doesn't contain the metastorage. Further, we may cast the result to
{code:java}
SnapshotFutureTaskResult res = (SnapshotFutureTaskResult)task0.result();
{code}
but it is a
{code:java}
Collections#EMPTY_SET
{code}
Stacktrace:
{code:java}
[ERROR][snapshot-runner-#413%dump.IgniteCacheDumpSelfTest0%][IgniteTestResources]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.ClassCastException: class
java.util.Collections$EmptySet cannot be cast to class
o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
(java.util.Collections$EmptySet is in module java.base of loader 'bootstrap';
o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in
unnamed module of loader 'app')]]
java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be
cast to class
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
(java.util.Collections$EmptySet is in module java.base of loader 'bootstrap';
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
is in unnamed module of loader 'app')
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1221)
~[classes/:?]
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180)
~[classes/:?]
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163)
~[classes/:?]
{code}
2) Even we fix the cast, another failure may arise. If cache with node filter
was created by a client or not by the excluded node (or may be not from the
coordinator), it may be checked before any dump data are written.
{code:java}
Caused by: class org.apache.ignite.IgniteException: Ouch! Argument is invalid:
binary metadata directory not exists
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:382)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.invoke(SnapshotPartitionsVerifyHandler.java:183)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:62)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:37)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invoke(IgniteSnapshotManager.java:3396)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.lambda$invokeAll$8d56ceed$1(IgniteSnapshotManager.java:3328)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11715)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11617)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11597)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invokeAll(IgniteSnapshotManager.java:3325)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1254)
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180)
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163)
... 3 more
Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid:
binary metadata directory not exists
at
org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:110)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.standaloneKernalContext(Dump.java:169)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.(Dump.java:149)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:370)
{code}
3) We might fix this with creating normal dump task instead of
{code:java}
task0 = new GridFinishedFuture<>(new
SnapshotFutureTaskResult(Collections.emptySet(), null))
{code}
It could write the nessesary dump content before checking it.
But one more fauilure is met:
{code:java}
Caused by: class org.apache.ignite.IgniteException: Wrong number of group
directories: 0
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpGroupDirectory(Dump.java:384)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.configs(Dump.java:237)
at org.apache.ignite.dump.DumpReader.lambda$run$2(DumpReader.java:107)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1785)
at
java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at
java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at
java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at
java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.AbstractCacheDumpTest$1.onCacheConfigs(AbstractCacheDumpTest.java:370)
at org.apache.ignite.dump.DumpReader.run(DumpReader.java:106)
... 12 more
{code}
There is no cache group directory at all on the filtered node.
was:
Ignite may fail to create dump of cache with a node filter.
1) If there are no partitions on some node, we assign the empty set as the
snapshot task result:
{code:java}
if (parts.isEmpty() && !withMetaStorage)
task0 = new GridFinishedFuture<>(Collections.emptySet());
else {
...
}
{code}
Cache dump doesn't contain the metastorage. Further, we may cast the result to
{code:java}
SnapshotFutureTaskResult res = (SnapshotFutureTaskResult)task0.result();
{code}
but it is a
{code:java}
Collections#EMPTY_SET
{code}
Stacktrace:
{code:java}
[ERROR][snapshot-runner-#413%dump.IgniteCacheDumpSelfTest0%][IgniteTestResources]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=java.lang.ClassCastException: class
java.util.Collections$EmptySet cannot be cast to class
o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
(java.util.Collections$EmptySet is in module java.base of loader 'bootstrap';
o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in
unnamed module of loader 'app')]]
java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be
cast to class
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
(java.util.Collections$EmptySet is in module java.base of loader 'bootstrap';
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
is in unnamed module of loader 'app')
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1221)
~[classes/:?]
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180)
~[classes/:?]
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163)
~[classes/:?]
{code}
2) Even we fix the cast, further failure appears:
{code:java}
private GridKernalContext standaloneKernalContext(File dumpDir, IgniteLogger
log) {
File binaryMeta =
CacheObjectBinaryProcessorImpl.binaryWorkDir(dumpDir.getAbsolutePath(),
F.first(metadata).folderName());
File marshaller = new File(dumpDir, DFLT_MARSHALLER_PATH);
A.ensure(binaryMeta.exists(), "binary metadata directory not exists");
{code}
If cache with node filter was created by a client or not by the excluded node
(or may be not from the coordinator), it may be checked before any dump data
are written.
Stacktrace:
{code:java}
Caused by: class org.apache.ignite.IgniteException: Ouch! Argument is invalid:
binary metadata directory not exists
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:382)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.invoke(SnapshotPartitionsVerifyHandler.java:183)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:62)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:37)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invoke(IgniteSnapshotManager.java:3396)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.lambda$invokeAll$8d56ceed$1(IgniteSnapshotManager.java:3328)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11715)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11617)
at
org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11597)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invokeAll(IgniteSnapshotManager.java:3325)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1254)
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180)
at
org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163)
... 3 more
Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid:
binary metadata directory not exists
at
org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:110)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.standaloneKernalContext(Dump.java:169)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.(Dump.java:149)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:370)
{code}
3) We might fix this with creating normal dump task instead of
{code:java}
task0 = new GridFinishedFuture<>(new
SnapshotFutureTaskResult(Collections.emptySet(), null))
{code}
It could write the nessesary dump content before checking it.
But one more fauilure is met:
{code:java}
Caused by: class org.apache.ignite.IgniteException: Wrong number of group
directories: 0
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpGroupDirectory(Dump.java:384)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.configs(Dump.java:237)
at org.apache.ignite.dump.DumpReader.lambda$run$2(DumpReader.java:107)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at java.base/java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1785)
at
java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
at
java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at
java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at
java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.AbstractCacheDumpTest$1.onCacheConfigs(AbstractCacheDumpTest.java:370)
at org.apache.ignite.dump.DumpReader.run(DumpReader.java:106)
... 12 more
{code}
There is no cache group directory at all on the filtered node.
> Cache dump fails on a cache with a node filter.
> -----------------------------------------------
>
> Key: IGNITE-21656
> URL: https://issues.apache.org/jira/browse/IGNITE-21656
> Project: Ignite
> Issue Type: Bug
> Reporter: Vladimir Steshin
> Priority: Major
> Labels: ise
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Ignite may fail to create dump of cache with a node filter.
> 1) If there are no partitions on some node, we assign the empty set as the
> snapshot task result:
> {code:java}
> if (parts.isEmpty() && !withMetaStorage)
> task0 = new GridFinishedFuture<>(Collections.emptySet());
> else {
> ...
> }
> {code}
> Cache dump doesn't contain the metastorage. Further, we may cast the result to
> {code:java}
> SnapshotFutureTaskResult res = (SnapshotFutureTaskResult)task0.result();
> {code}
> but it is a
> {code:java}
> Collections#EMPTY_SET
> {code}
> Stacktrace:
> {code:java}
> [ERROR][snapshot-runner-#413%dump.IgniteCacheDumpSelfTest0%][IgniteTestResources]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
> SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
> [type=CRITICAL_ERROR, err=java.lang.ClassCastException: class
> java.util.Collections$EmptySet cannot be cast to class
> o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
> (java.util.Collections$EmptySet is in module java.base of loader 'bootstrap';
> o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in
> unnamed module of loader 'app')]]
> java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be
> cast to class
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
> (java.util.Collections$EmptySet is in module java.base of loader
> 'bootstrap';
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult
> is in unnamed module of loader 'app')
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1221)
> ~[classes/:?]
> at
> org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180)
> ~[classes/:?]
> at
> org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163)
> ~[classes/:?]
> {code}
> 2) Even we fix the cast, another failure may arise. If cache with node filter
> was created by a client or not by the excluded node (or may be not from the
> coordinator), it may be checked before any dump data are written.
> {code:java}
> Caused by: class org.apache.ignite.IgniteException: Ouch! Argument is
> invalid: binary metadata directory not exists
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:382)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.invoke(SnapshotPartitionsVerifyHandler.java:183)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:62)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:37)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invoke(IgniteSnapshotManager.java:3396)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.lambda$invokeAll$8d56ceed$1(IgniteSnapshotManager.java:3328)
> at
> org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11715)
> at
> org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11617)
> at
> org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11597)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invokeAll(IgniteSnapshotManager.java:3325)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1254)
> at
> org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180)
> at
> org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163)
> ... 3 more
> Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid:
> binary metadata directory not exists
> at
> org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:110)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.standaloneKernalContext(Dump.java:169)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.(Dump.java:149)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:370)
> {code}
> 3) We might fix this with creating normal dump task instead of
> {code:java}
> task0 = new GridFinishedFuture<>(new
> SnapshotFutureTaskResult(Collections.emptySet(), null))
> {code}
> It could write the nessesary dump content before checking it.
> But one more fauilure is met:
> {code:java}
> Caused by: class org.apache.ignite.IgniteException: Wrong number of group
> directories: 0
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpGroupDirectory(Dump.java:384)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.configs(Dump.java:237)
> at org.apache.ignite.dump.DumpReader.lambda$run$2(DumpReader.java:107)
> at
> java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
> at java.base/java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1785)
> at
> java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
> at
> java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
> at
> java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
> at
> java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
> at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
> at
> org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.AbstractCacheDumpTest$1.onCacheConfigs(AbstractCacheDumpTest.java:370)
> at org.apache.ignite.dump.DumpReader.run(DumpReader.java:106)
> ... 12 more
> {code}
> There is no cache group directory at all on the filtered node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
