[
https://issues.apache.org/jira/browse/IGNITE-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982473#comment-16982473
]
Anton Kalashnikov commented on IGNITE-11939:
--------------------------------------------
I figured out the following things:
First of all, we are trying to release a segment from two places(exchange and
full message handler):
{noformat}
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:101)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:219)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:997)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java
:1865)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processFullPartitionUpdate(GridCachePartitionExchangeManager.java:1829)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:441)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:428)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3622)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3601)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:101)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:219)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:997)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java:1865)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1438)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:882)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3172)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[16:35:58]W: [org.apache.ignite:ignite-indexing]
java.lang.Thread.run(Thread.java:748)
{noformat}
But as I can see method
GridCacheDatabaseSharedManager#releaseHistoryForPreloading doesn't design for
concurrency usage due to the count of iteration through reservedForPreloading
will be unpredictable. So if add synchronization to this method it'll fix that
problem(https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PdsIndexing?branch=pull%2F7066%2Fhead&buildTypeTab=overview)
Take a look at releaseHistoryLock bellow:
{noformat}
private final ReentrantLock releaseHistoryLock = new ReentrantLock();
/** {@inheritDoc} */
@Override public void releaseHistoryForPreloading() {
releaseHistoryLock.lock();
try {
for (Map.Entry<T2<Integer, Integer>, T2<Long, WALPointer>> e :
reservedForPreloading.entrySet()) {
try {
cctx.wal().release(e.getValue().get2());
}
catch (IgniteCheckedException ex) {
U.error(log, "Could not release WAL reservation", ex);
throw new IgniteException(ex);
}
}
reservedForPreloading.clear();
}
finally {
releaseHistoryLock.unlock();
}
}
{noformat}
[~ivan.glukos] Can you help with the next steps to solve this problem?
> IgnitePdsTxHistoricalRebalancingTest.testTopologyChangesWithConstantLoad
> test failure
> --------------------------------------------------------------------------------------
>
> Key: IGNITE-11939
> URL: https://issues.apache.org/jira/browse/IGNITE-11939
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexei Scherbakov
> Assignee: Anton Kalashnikov
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Caused by exception on releasing reserved segments:
> {noformat}
> [12:51:23]W: [org.apache.ignite:ignite-indexing] [2019-06-21
> 12:51:23,967][ERROR][exchange-worker-#33825%persistence.IgnitePdsTxHistoricalRebalancingTest1%][GridDhtPartitionsExchangeFuture]
> Failed to reinitialize local partitions (rebalancing will be stopped)
> : GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7,
> minorTopVer=1], discoEvt=DiscoveryCustomEvent
> [customMsg=CacheAffinityChangeMessage
> [id=08de0ff7b61-276ac575-e4dc-4525-b24b-d0a5d1d7633d,
> topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], exc
> hId=null, partsMsg=null, exchangeNeeded=true],
> affTopVer=AffinityTopologyVersion [topVer=7, minorTopVer=1],
> super=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=97e46568-6aa0-4a4b-864c-f05415c00000,
> consistentId=persistence.IgnitePdsTxHistoricalRebalancingTest0, addrs=Arra
> yList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500,
> order=1, intOrder=1, lastExchangeTime=1561110643882, loc=false,
> ver=2.8.0#20190621-sha1:00000000, isClient=false], topVer=7,
> nodeId8=0ff3354e, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=15611106839
> 58]], nodeId=97e46568, evt=DISCOVERY_CUSTOM_EVT]
> [12:51:23]W: [org.apache.ignite:ignite-indexing]
> java.lang.AssertionError: cur=null, absIdx=0
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:55)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:207)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:983)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java:1844)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1431)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:862)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3079)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2928)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> java.lang.Thread.run(Thread.java:748)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] [12:51:23] (err)
> Failed to notify listener:
> o.a.i.i.processors.timeout.GridTimeoutProcessor$2...@79ba1907java.lang.AssertionError:
> cur=null, absIdx=0
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:55)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:207)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:983)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java:1844)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1431)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:862)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3079)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2928)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [12:51:23]W: [org.apache.ignite:ignite-indexing] at
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> Timed out run link [1]
> [1]
> https://ci.ignite.apache.org/viewLog.html?buildId=4168883&buildTypeId=IgniteTests24Java8_PdsIndexing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
