[jira] [Updated] (IGNITE-9004) Failed to move temp file during segment creation
[
https://issues.apache.org/jira/browse/IGNITE-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Kalashnikov updated IGNITE-9004:
--
Description:
Reproduced by Activate/Deactivate suit, almost any tests in
IgniteChangeGlobalStateTest class. for example
IgniteChangeGlobalStateTest#testStopPrimaryAndActivateFromClientNode
{noformat}
Failed to reinitialize local partitions (preloading will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6,
minorTopVer=1], discoEvt=DiscoveryCustomEvent
[customMsg=ChangeGlobalStateMessage
[id=9093c48a461-165cdacd-8a3b-4072-9f48-e80e1b63fda9,
reqId=07393ea5-1c6a-4581-b016-9eb88d6bd978,
initiatingNodeId=8dced5ba-725d-494b-8e8e-ffc76453fecd, activate=true,
baselineTopology=BaselineTopology [id=0, branchingHash=314980173,
branchingType='Cluster activation', baselineNodes=[node2, node0, node1]],
forceChangeBaselineTopology=false, timestamp=1531832492029],
affTopVer=AffinityTopologyVersion [topVer=6, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=8dced5ba-725d-494b-8e8e-ffc76453fecd, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1,
172.25.4.132], sockAddrs=[/172.25.4.132:47504, /0:0:0:0:0:0:0:1%lo:47504,
/127.0.0.1:47504], discPort=47504, order=2, intOrder=2,
lastExchangeTime=1531832486546, loc=false, ver=2.7.0#19700101-sha1:,
isClient=false], topVer=6, nodeId8=9960f6b9, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1531832492035]], nodeId=8dced5ba,
evt=DISCOVERY_CUSTOM_EVT]
java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:358)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:400)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:384)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:783)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:627)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:144)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:169)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.(AbstractFreeList.java:371)
at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage$FreeListImpl.(MetaStorage.java:484)
at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.init(MetaStorage.java:143)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:852)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:954)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:661)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2484)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2364)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Given:
# Activated Node1-1 in grid1.
# MetaStorage on node1-1 in OffHeap.
# MetaStorage have not storage on disk yet.
When:
# Checkpoint on node1-1 is starting. Start checkpoint marker was written.
# node2-1 in grid2 is starting.(grid1 and grid2 have same persistence)
Then:
# node2-1 found expected checkpoint marker("Found unexpected checkpoint
marker") and initialize FilePageStore for metaStorage by empty page
# node1-1 finished checkpoint and wrote MetaStorage on disk.
# After stop grid1 and activate grid2 node2-1 was failed because try read more
than one page.
Possible solution:
* We can skip initialization FilePageStore for MetaStorage by empty page during
the start
* We can take a lock for metaStorage that only one node can read or write one
MetaStorage in one moment.
* We can reinitialize FilePageStore from disk when we activate cluster.
was:
Reproduced by Activate/Deactivate suit, for example
IgniteChangeGlobalStateTest#testStopPrimaryAndActivateFromClientNode
{noformat}
class org.apache.ignite.internal.pagemem.wal.StorageException: Failed to move
temp file to a regular WAL segment file: /data/teamcity/work/c182b70f2dfa650
7/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal
[13:56:05]W: [org.apache.ignite:ignite-core]
[jira] [Updated] (IGNITE-9004) Failed to move temp file during segment creation
[
https://issues.apache.org/jira/browse/IGNITE-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy Pavlov updated IGNITE-9004:
---
Priority: Critical (was: Major)
> Failed to move temp file during segment creation
>
>
> Key: IGNITE-9004
> URL: https://issues.apache.org/jira/browse/IGNITE-9004
> Project: Ignite
> Issue Type: Test
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Critical
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.7
>
>
> Reproduced by Activate/Deactivate suit, for example
> IgniteChangeGlobalStateTest#testStopPrimaryAndActivateFromClientNode
> {noformat}
> class org.apache.ignite.internal.pagemem.wal.StorageException: Failed to move
> temp file to a regular WAL segment file: /data/teamcity/work/c182b70f2dfa650
> 7/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.createFile(FileWriteAheadLogManager.java:1446)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkFiles(FileWriteAheadLogManager.java:2269)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.access$4500(FileWriteAheadLogManager.java:143)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.allocateRemainingFiles(FileWriteAheadLogManage
> r.java:1862)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1606)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> java.lang.Thread.run(Thread.java:748)
> [13:56:05]W: [org.apache.ignite:ignite-core] Caused by:
> java.nio.file.NoSuchFileException:
> /data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal.tmp
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> java.nio.file.Files.move(Files.java:1395)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.createFile(FileWriteAheadLogManager.java:1442)
> [13:56:05]W: [org.apache.ignite:ignite-core] ... 6 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IGNITE-9004) Failed to move temp file during segment creation
[
https://issues.apache.org/jira/browse/IGNITE-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy Pavlov updated IGNITE-9004:
---
Labels: MakeTeamcityGreenAgain (was: )
> Failed to move temp file during segment creation
>
>
> Key: IGNITE-9004
> URL: https://issues.apache.org/jira/browse/IGNITE-9004
> Project: Ignite
> Issue Type: Test
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.7
>
>
> Reproduced by Activate/Deactivate suit, for example
> IgniteChangeGlobalStateTest#testStopPrimaryAndActivateFromClientNode
> {noformat}
> class org.apache.ignite.internal.pagemem.wal.StorageException: Failed to move
> temp file to a regular WAL segment file: /data/teamcity/work/c182b70f2dfa650
> 7/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.createFile(FileWriteAheadLogManager.java:1446)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkFiles(FileWriteAheadLogManager.java:2269)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.access$4500(FileWriteAheadLogManager.java:143)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.allocateRemainingFiles(FileWriteAheadLogManage
> r.java:1862)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1606)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> java.lang.Thread.run(Thread.java:748)
> [13:56:05]W: [org.apache.ignite:ignite-core] Caused by:
> java.nio.file.NoSuchFileException:
> /data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal.tmp
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> java.nio.file.Files.move(Files.java:1395)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.createFile(FileWriteAheadLogManager.java:1442)
> [13:56:05]W: [org.apache.ignite:ignite-core] ... 6 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (IGNITE-9004) Failed to move temp file during segment creation
[
https://issues.apache.org/jira/browse/IGNITE-9004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy Pavlov updated IGNITE-9004:
---
Fix Version/s: 2.7
> Failed to move temp file during segment creation
>
>
> Key: IGNITE-9004
> URL: https://issues.apache.org/jira/browse/IGNITE-9004
> Project: Ignite
> Issue Type: Test
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Critical
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.7
>
>
> Reproduced by Activate/Deactivate suit, for example
> IgniteChangeGlobalStateTest#testStopPrimaryAndActivateFromClientNode
> {noformat}
> class org.apache.ignite.internal.pagemem.wal.StorageException: Failed to move
> temp file to a regular WAL segment file: /data/teamcity/work/c182b70f2dfa650
> 7/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.createFile(FileWriteAheadLogManager.java:1446)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkFiles(FileWriteAheadLogManager.java:2269)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.access$4500(FileWriteAheadLogManager.java:143)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.allocateRemainingFiles(FileWriteAheadLogManage
> r.java:1862)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1606)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> java.lang.Thread.run(Thread.java:748)
> [13:56:05]W: [org.apache.ignite:ignite-core] Caused by:
> java.nio.file.NoSuchFileException:
> /data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/wal/node1/0002.wal.tmp
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> java.nio.file.Files.move(Files.java:1395)
> [13:56:05]W: [org.apache.ignite:ignite-core] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.createFile(FileWriteAheadLogManager.java:1442)
> [13:56:05]W: [org.apache.ignite:ignite-core] ... 6 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
