[jira] [Commented] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409914#comment-17409914
]
Sarunas Valaskevicius commented on IGNITE-12850:
Hi, this marks it as fixed in 2.8, but I'm pretty sure my colleague reproduced
it in 2.10 as well. He confirmed that persistence did cause ignite getting
stuck for him, albeit cannot say if the stack trace was the same.
I won't reopen it, but please can you test as described above? (not just nice
disconnects as in the test, but random process kills too)
Maybe other people who reproduced it could comment with the version number too?
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8, 2.7.6
>Reporter: Sarunas Valaskevicius
>Assignee: Sergei Ryzhov
>Priority: Blocker
> Labels: IEP-56, ise
> Fix For: 2.8
>
> Attachments: TestRestoreHistory.java
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates base line topology history with gaps
> # from that point, it is impossible to start the node as
> `{color:#569e16}restoreHistory{color}` throws an exception when it is
> processing the gap
> –
> tested on 2.7.6, but it seems that ignite 2.8.0 would suffer from the same
> issue - by looking at the code
> --
> {code:java}
> 2020-03-21_00:00:03.867 [fapi-main-0] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
> BaselineTopology[id=9]
> 2020-03-21_00:00:03.904 [fapi-main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology
> history has failed, expected history item not found for id=8
> at
> org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408714#comment-17408714
]
Sarunas Valaskevicius commented on IGNITE-12850:
Hi. IIRC I found it during manual testing before the upgrade, and the steps had
been roughly:
# Have a 3 node cluster with persistence on
# keep restarting nodes randomly (even killing the processes randomly)
In this case I was not waiting for other nodes to recover, just hammer with
restarts (with a short random delay) and see that in the end, the cluster is
able to get back up.
Note : I didn't need to run such a script for long to find it, but my colleague
had to run for quite a while.
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8, 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates base line topology history with gaps
> # from that point, it is impossible to start the node as
> `{color:#569e16}restoreHistory{color}` throws an exception when it is
> processing the gap
> –
> tested on 2.7.6, but it seems that ignite 2.8.0 would suffer from the same
> issue - by looking at the code
> --
> {code:java}
> 2020-03-21_00:00:03.867 [fapi-main-0] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
> BaselineTopology[id=9]
> 2020-03-21_00:00:03.904 [fapi-main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology
> history has failed, expected history item not found for id=8
> at
> org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12850:
---
Description:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history item, and generates base line topology history with gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
–
tested on 2.7.6, but it seems that ignite 2.8.0 would suffer from the same
issue - by looking at the code
--
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
was:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history item, and generates base line topology history with gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
–
it seems that ignite 2.8.0 would suffer from the same issue - by looking at
the code
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8, 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
>
[jira] [Updated] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12850:
---
Affects Version/s: 2.8
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8, 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates base line topology history with gaps
> # from that point, it is impossible to start the node as
> `{color:#569e16}restoreHistory{color}` throws an exception when it is
> processing the gap
> –
> it seems that ignite 2.8.0 would suffer from the same issue - by looking at
> the code
> {code:java}
> 2020-03-21_00:00:03.867 [fapi-main-0] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
> BaselineTopology[id=9]
> 2020-03-21_00:00:03.904 [fapi-main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology
> history has failed, expected history item not found for id=8
> at
> org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072082#comment-17072082
]
Sarunas Valaskevicius edited comment on IGNITE-12850 at 3/31/20, 7:24 PM:
--
Please can you also suggest a short-term workaround as well as a proper fix?
I found that if I delete metastorage persistence directory ignite starts ok -
why is it hardcoded to persistence true? what problems would I see if I deleted
the metastorage before starting the node?
was (Author: sarunas):
Please can you also suggest a short-term workaround as well as a proper fix?
I found that if I delete metastorage persistence directory ignite starts ok -
why is it hardcoded to persistence true? what problems would I see if I delete
metastorage before starting the node?
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates base line topology history with gaps
> # from that point, it is impossible to start the node as
> `{color:#569e16}restoreHistory{color}` throws an exception when it is
> processing the gap
> –
> it seems that ignite 2.8.0 would suffer from the same issue - by looking at
> the code
> {code:java}
> 2020-03-21_00:00:03.867 [fapi-main-0] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
> BaselineTopology[id=9]
> 2020-03-21_00:00:03.904 [fapi-main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology
> history has failed, expected history item not found for id=8
> at
> org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072082#comment-17072082
]
Sarunas Valaskevicius commented on IGNITE-12850:
Please can you also suggest a short-term workaround as well as a proper fix?
I found that if I delete metastorage persistence directory ignite starts ok -
why is it hardcoded to persistence true? what problems would I see if I delete
metastorage before starting the node?
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates base line topology history with gaps
> # from that point, it is impossible to start the node as
> `{color:#569e16}restoreHistory{color}` throws an exception when it is
> processing the gap
> –
> it seems that ignite 2.8.0 would suffer from the same issue - by looking at
> the code
> {code:java}
> 2020-03-21_00:00:03.867 [fapi-main-0] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
> BaselineTopology[id=9]
> 2020-03-21_00:00:03.904 [fapi-main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology
> history has failed, expected history item not found for id=8
> at
> org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12850:
---
Description:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history item, and generates base line topology history with gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
–
it seems that ignite 2.8.0 would suffer from the same issue - by looking at
the code
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
was:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history item, and generates base line topology with history in gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
–
it seems that ignite 2.8.0 would suffer from the same issue - by looking at
the code
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates
[jira] [Updated] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12850:
---
Description:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history item, and generates base line topology with history in gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
–
it seems that ignite 2.8.0 would suffer from the same issue - by looking at
the code
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
was:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history, and generates base line topology with history in gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
--
it seems that ignite 2.8.0 would suffer from the same issue - by looking at the
code
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history item, and generates
[jira] [Updated] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12850:
---
Description:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history, and generates base line topology with history in gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
--
it seems that ignite 2.8.0 would suffer from the same issue - by looking at the
code
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
was:
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history, and generates base line topology with history in gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history, and generates base line topology with history in gaps
> # from that point, it is impossible to start the node as
[jira] [Updated] (IGNITE-12850) Ignite node cannot be started (metastorage history loading fails)
[
https://issues.apache.org/jira/browse/IGNITE-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12850:
---
Summary: Ignite node cannot be started (metastorage history loading fails)
(was: Ignite cannot start (metastorage history loading fails))
> Ignite node cannot be started (metastorage history loading fails)
> -
>
> Key: IGNITE-12850
> URL: https://issues.apache.org/jira/browse/IGNITE-12850
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.7.6
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> # metastorage is using persistence
> # when a node is ready to write, writeBaselineTopology is called with null
> history, and generates base line topology with history in gaps
> # from that point, it is impossible to start the node as
> `{color:#569e16}restoreHistory{color}` throws an exception when it is
> processing the gap
> {code:java}
> 2020-03-21_00:00:03.867 [fapi-main-0] INFO
> o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
> BaselineTopology[id=9]
> 2020-03-21_00:00:03.904 [fapi-main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology
> history has failed, expected history item not found for id=8
> at
> org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
> at
> org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12850) Ignite cannot start (metastorage history loading fails)
Sarunas Valaskevicius created IGNITE-12850:
--
Summary: Ignite cannot start (metastorage history loading fails)
Key: IGNITE-12850
URL: https://issues.apache.org/jira/browse/IGNITE-12850
Project: Ignite
Issue Type: Bug
Components: cache
Affects Versions: 2.7.6
Reporter: Sarunas Valaskevicius
# metastorage is using persistence
# when a node is ready to write, writeBaselineTopology is called with null
history, and generates base line topology with history in gaps
# from that point, it is impossible to start the node as
`{color:#569e16}restoreHistory{color}` throws an exception when it is
processing the gap
{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history
has failed, expected history item not found for id=8
at
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12805) Node fails to restart
[
https://issues.apache.org/jira/browse/IGNITE-12805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063601#comment-17063601
]
Sarunas Valaskevicius commented on IGNITE-12805:
Hi [~slava.koptilin] ,
yes, the idea is that ignite would pull data from the remote storage on demand,
and start managing it from that point onwards within the service context,
pushing it back for sharing (across services) and backup purposes. the config:
{code:java}
val cfg = new CacheConfiguration[String, BinaryObject](s"name")
cfg.setBackups(1)
cfg.setCacheMode(CacheMode.PARTITIONED)
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC)
cfg.setGroupName(s"name")
cfg.setDataRegionName("name")
cfg.setReadThrough(true)
cfg.setCacheLoaderFactory(new CachedEntityStorage.CacheLoaderFactory(...))
cfg.setWriteThrough(true)
cfg.setCacheWriterFactory(new CachedEntityStorage.CacheWriterFactory(...))
cfg.setWriteBehindEnabled(true)
cfg.setWriteBehindCoalescing(true)
cfg.setWriteBehindFlushSize(0)
cfg.setWriteBehindFlushFrequency(1000)
cfg.setWriteBehindFlushThreadCount(2)
cfg.setWriteBehindBatchSize(32)
cfg.setStoreKeepBinary(true)
{code}
the region is configured with persistence=true on all ignite server nodes
Thanks for looking into this!
> Node fails to restart
> -
>
> Key: IGNITE-12805
> URL: https://issues.apache.org/jira/browse/IGNITE-12805
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8
>Reporter: Sarunas Valaskevicius
>Assignee: Vyacheslav Koptilin
>Priority: Blocker
>
> 1. nodes have default persistence false, but there is a cache region with
> persistence on.
> 2. a cluster starts ok with ignite data directory clean
> 3. but when the nodes are restarted, they fail and can never join the cluster
> again:
>
> {code:java}
> 12:352020-03-19_13:34:30.273 [main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> java.lang.NullPointerException: null
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
> at
> org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-12805) Node fails to restart
[
https://issues.apache.org/jira/browse/IGNITE-12805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063601#comment-17063601
]
Sarunas Valaskevicius edited comment on IGNITE-12805 at 3/20/20, 7:49 PM:
--
Hi [~slava.koptilin] ,
yes, the idea is that ignite would pull data from the remote storage on demand
(first use), and start managing it from that point onwards within the service
context, pushing it back for sharing (across services) and backup purposes. the
config:
{code:java}
val cfg = new CacheConfiguration[String, BinaryObject](s"name")
cfg.setBackups(1)
cfg.setCacheMode(CacheMode.PARTITIONED)
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC)
cfg.setGroupName(s"name")
cfg.setDataRegionName("name")
cfg.setReadThrough(true)
cfg.setCacheLoaderFactory(new CachedEntityStorage.CacheLoaderFactory(...))
cfg.setWriteThrough(true)
cfg.setCacheWriterFactory(new CachedEntityStorage.CacheWriterFactory(...))
cfg.setWriteBehindEnabled(true)
cfg.setWriteBehindCoalescing(true)
cfg.setWriteBehindFlushSize(0)
cfg.setWriteBehindFlushFrequency(1000)
cfg.setWriteBehindFlushThreadCount(2)
cfg.setWriteBehindBatchSize(32)
cfg.setStoreKeepBinary(true)
{code}
the region is configured with persistence=true on all ignite server nodes
Thanks for looking into this!
was (Author: sarunas):
Hi [~slava.koptilin] ,
yes, the idea is that ignite would pull data from the remote storage on demand,
and start managing it from that point onwards within the service context,
pushing it back for sharing (across services) and backup purposes. the config:
{code:java}
val cfg = new CacheConfiguration[String, BinaryObject](s"name")
cfg.setBackups(1)
cfg.setCacheMode(CacheMode.PARTITIONED)
cfg.setAtomicityMode(CacheAtomicityMode.ATOMIC)
cfg.setGroupName(s"name")
cfg.setDataRegionName("name")
cfg.setReadThrough(true)
cfg.setCacheLoaderFactory(new CachedEntityStorage.CacheLoaderFactory(...))
cfg.setWriteThrough(true)
cfg.setCacheWriterFactory(new CachedEntityStorage.CacheWriterFactory(...))
cfg.setWriteBehindEnabled(true)
cfg.setWriteBehindCoalescing(true)
cfg.setWriteBehindFlushSize(0)
cfg.setWriteBehindFlushFrequency(1000)
cfg.setWriteBehindFlushThreadCount(2)
cfg.setWriteBehindBatchSize(32)
cfg.setStoreKeepBinary(true)
{code}
the region is configured with persistence=true on all ignite server nodes
Thanks for looking into this!
> Node fails to restart
> -
>
> Key: IGNITE-12805
> URL: https://issues.apache.org/jira/browse/IGNITE-12805
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8
>Reporter: Sarunas Valaskevicius
>Assignee: Vyacheslav Koptilin
>Priority: Blocker
>
> 1. nodes have default persistence false, but there is a cache region with
> persistence on.
> 2. a cluster starts ok with ignite data directory clean
> 3. but when the nodes are restarted, they fail and can never join the cluster
> again:
>
> {code:java}
> 12:352020-03-19_13:34:30.273 [main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connections
> java.lang.NullPointerException: null
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
> at
> org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
>
[jira] [Updated] (IGNITE-12805) Node fails to restart
[
https://issues.apache.org/jira/browse/IGNITE-12805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12805:
---
Description:
1. nodes have default persistence false, but there is a cache region with
persistence on.
2. a cluster starts ok with ignite data directory clean
3. but when the nodes are restarted, they fail and can never join the cluster
again:
{code:java}
12:352020-03-19_13:34:30.273 [main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connectionsjava.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
at
org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637)
{code}
was:
1. nodes have default persistence false, but there is a cache region with
persistence on.
2. a cluster starts ok with ignite data directory clean
3. but when the nodes are restarted, they fail and can never join the cluster
again:
{code:java}
12:352020-03-19_13:34:30.273 [main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connectionsjava.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
at
org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637) {code}
> Node fails to restart
> -
>
> Key: IGNITE-12805
> URL: https://issues.apache.org/jira/browse/IGNITE-12805
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8
>
[jira] [Updated] (IGNITE-12805) Node fails to restart
[
https://issues.apache.org/jira/browse/IGNITE-12805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12805:
---
Description:
1. nodes have default persistence false, but there is a cache region with
persistence on.
2. a cluster starts ok with ignite data directory clean
3. but when the nodes are restarted, they fail and can never join the cluster
again:
{code:java}
12:352020-03-19_13:34:30.273 [main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connections
java.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
at
org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637)
{code}
was:
1. nodes have default persistence false, but there is a cache region with
persistence on.
2. a cluster starts ok with ignite data directory clean
3. but when the nodes are restarted, they fail and can never join the cluster
again:
{code:java}
12:352020-03-19_13:34:30.273 [main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connectionsjava.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
at
org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637)
{code}
> Node fails to restart
> -
>
> Key: IGNITE-12805
> URL: https://issues.apache.org/jira/browse/IGNITE-12805
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8
>
[jira] [Updated] (IGNITE-12805) Node fails to restart
[
https://issues.apache.org/jira/browse/IGNITE-12805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarunas Valaskevicius updated IGNITE-12805:
---
Priority: Blocker (was: Major)
> Node fails to restart
> -
>
> Key: IGNITE-12805
> URL: https://issues.apache.org/jira/browse/IGNITE-12805
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.8
>Reporter: Sarunas Valaskevicius
>Priority: Blocker
>
> 1. nodes have default persistence false, but there is a cache region with
> persistence on.
> 2. a cluster starts ok with ignite data directory clean
> 3. but when the nodes are restarted, they fail and can never join the cluster
> again:
>
> {code:java}
> 12:352020-03-19_13:34:30.273 [main-0] ERROR
> o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
> node will be stopped and close connectionsjava.lang.NullPointerException:
> nullat
> org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
> at
> org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
> at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
>at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12805) Node fails to restart
Sarunas Valaskevicius created IGNITE-12805:
--
Summary: Node fails to restart
Key: IGNITE-12805
URL: https://issues.apache.org/jira/browse/IGNITE-12805
Project: Ignite
Issue Type: Bug
Components: cache
Affects Versions: 2.8
Reporter: Sarunas Valaskevicius
1. nodes have default persistence false, but there is a cache region with
persistence on.
2. a cluster starts ok with ignite data directory clean
3. but when the nodes are restarted, they fail and can never join the cluster
again:
{code:java}
12:352020-03-19_13:34:30.273 [main-0] ERROR
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors,
node will be stopped and close connectionsjava.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.affinityNode(GridCacheUtils.java:1374)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$CachePredicate.dataNode(GridDiscoveryManager.java:3205)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheAffinityNode(GridDiscoveryManager.java:1894)
at
org.apache.ignite.internal.processors.cache.ValidationOnNodeJoinUtils.validate(ValidationOnNodeJoinUtils.java:330)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCacheContext(GridCacheProcessor.java:1201)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2291)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1700(GridCacheProcessor.java:202)
at
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5387)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:1075)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:2068)
at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1254)at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
