[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379945#comment-17379945 ] Pavel Tupitsyn commented on IGNITE-11704: - TC run above is for https://github.com/apache/ignite/pull/9249: "IGNITE-11704 Add PARTITION_META_PAGE_DELTA_RECORD_V4 WALRecord type placeholder" > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Alexey Scherbakov >Priority: Major > Labels: rebalance > Time Spent: 40m > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379825#comment-17379825
]
Ignite TC Bot commented on IGNITE-11704:
{panel:title=Branch: [pull/9249/head] Base: [master] : Possible Blockers
(9)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Control Utility (Zookeeper){color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=6084190]]
{color:#d04437}Platform .NET (Long Running){color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=6084174]]
* exe: CacheTest.TestCacheWithExpiryPolicyOnAccess - Test has low fail rate in
base branch 0,0% and is not flaky
{color:#d04437}Basic 1{color} [[tests
2|https://ci.ignite.apache.org/viewLog.html?buildId=6084136]]
* IgniteBasicTestSuite: WALRecordTest.testAllTestWalRecordBuilderConfigured -
Test has low fail rate in base branch 0,0% and is not flaky
* IgniteBasicTestSuite:
WALRecordSerializationTest.testAllWalRecordsSerializedAndDeserializedSuccessfully
- Test has low fail rate in base branch 0,0% and is not flaky
{color:#d04437}Platform .NET (Core Linux){color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=6084172]]
{color:#d04437}Java Client{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=6084115]]
{color:#d04437}Cache (Failover) 1{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=6084144]]
{color:#d04437}Platform .NET{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=6084171]]
* exe: DataStorageMetricsTest.TestDataStorageMetrics - Test has low fail rate
in base branch 0,0% and is not flaky
{color:#d04437}Thin client: PHP{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=6084193]]
{panel}
{panel:title=Branch: [pull/9249/head] Base: [master] : New Tests
(3)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS (Indexing){color} [[tests
3|https://ci.ignite.apache.org/viewLog.html?buildId=6084165]]
* {color:#013220}IgnitePdsWithIndexingTestSuite:
RenameIndexTreeTest.testNotPersistRenamingIndexRootPage - PASSED{color}
* {color:#013220}IgnitePdsWithIndexingTestSuite:
RenameIndexTreeTest.testPersistRenamingIndexRootPage - PASSED{color}
* {color:#013220}IgnitePdsWithIndexingTestSuite:
RenameIndexTreeTest.testRenamingIndexRootPage - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6084197buildTypeId=IgniteTests24Java8_RunAll]
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Alexey Scherbakov
>Priority: Major
> Labels: rebalance
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an incomplete
> baseline).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230068#comment-17230068 ] Alexey Scherbakov commented on IGNITE-11704: [~mmuzaf] I have plans to merge it in early december. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Alexey Scherbakov >Priority: Major > Labels: rebalance > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229918#comment-17229918 ] Maxim Muzafarov commented on IGNITE-11704: -- [~ascherbakov] Hello, should we expect this issue to be included into 2.10 release? > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Alexey Scherbakov >Priority: Major > Labels: rebalance > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988954#comment-16988954
]
Alexei Scherbakov commented on IGNITE-11704:
Better to store a full log in case of TC history cleanup.
{noformat}
java.lang.AssertionError: Failed to wait for tombstone cleanup:
distributed.CacheRemoveWithTombstonesLoadTest2 expected:<0> but was:<1>
at
org.apache.ignite.internal.processors.cache.distributed.CacheRemoveWithTombstonesLoadTest.waitTombstoneCleanup(CacheRemoveWithTombstonesLoadTest.java:335)
at
org.apache.ignite.internal.processors.cache.distributed.CacheRemoveWithTombstonesLoadTest.removeAndRebalance(CacheRemoveWithTombstonesLoadTest.java:250)
--- Stdout: ---
[12:28:21]__
[12:28:21] / _/ ___/ |/ / _/_ __/ __/
[12:28:21] _/ // (7 7// / / / / _/
[12:28:21] /___/\___/_/|_/___/ /_/ /___/
[12:28:21]
[12:28:21] ver. 2.8.0-SNAPSHOT#20191203-sha1:DEV
[12:28:21] 2019 Copyright(C) Apache Software Foundation
[12:28:21]
[12:28:21] Ignite documentation: http://ignite.apache.org
[12:28:21]
[12:28:21] Quiet mode.
[12:28:21] ^-- Logging by 'GridTestLog4jLogger [quiet=true, config=null]'
[12:28:21] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[12:28:21]
[12:28:21] OS: Linux 4.15.0-54-generic amd64
[12:28:21] VM information: Java(TM) SE Runtime Environment 1.8.0_212-b10 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.212-b10
[12:28:21] Configured plugins:
[12:28:21] ^-- StanByClusterTestProvider 1.0
[12:28:21] ^-- null
[12:28:21]
[12:28:21] ^-- PageMemory tracker plugin 1.0
[12:28:21] ^--
[12:28:21]
[12:28:21] ^-- TestDistibutedConfigurationPlugin 1.0
[12:28:21] ^--
[12:28:21]
[12:28:21] ^-- NodeValidationPluginProvider 1.0
[12:28:21] ^--
[12:28:21]
[12:28:21] Configured failure handler: [hnd=NoOpFailureHandler
[super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT
[12:28:21] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[12:28:21] Security status [authentication=off, tls/ssl=off]
[12:28:21] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[12:28:21] Data Regions Configured:
[12:28:21] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB,
persistence=false, lazyMemoryAllocation=true]
[12:28:21]
[12:28:21] Ignite node started OK (id=a34d6c79, instance
name=distributed.CacheRemoveWithTombstonesLoadTest0)
[12:28:21] Topology snapshot [ver=1, locNode=a34d6c79, servers=1, clients=0,
state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB]
[12:28:23]__
[12:28:23] / _/ ___/ |/ / _/_ __/ __/
[12:28:23] _/ // (7 7// / / / / _/
[12:28:23] /___/\___/_/|_/___/ /_/ /___/
[12:28:23]
[12:28:23] ver. 2.8.0-SNAPSHOT#20191203-sha1:DEV
[12:28:23] 2019 Copyright(C) Apache Software Foundation
[12:28:23]
[12:28:23] Ignite documentation: http://ignite.apache.org
[12:28:23]
[12:28:23] Quiet mode.
[12:28:23] ^-- Logging by 'GridTestLog4jLogger [quiet=true, config=null]'
[12:28:23] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[12:28:23]
[12:28:23] OS: Linux 4.15.0-54-generic amd64
[12:28:23] VM information: Java(TM) SE Runtime Environment 1.8.0_212-b10 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.212-b10
[12:28:23] Configured plugins:
[12:28:23] ^-- StanByClusterTestProvider 1.0
[12:28:23] ^-- null
[12:28:23]
[12:28:23] ^-- PageMemory tracker plugin 1.0
[12:28:23] ^--
[12:28:23]
[12:28:23] ^-- TestDistibutedConfigurationPlugin 1.0
[12:28:23] ^--
[12:28:23]
[12:28:23] ^-- NodeValidationPluginProvider 1.0
[12:28:23] ^--
[12:28:23]
[12:28:23] Configured failure handler: [hnd=NoOpFailureHandler
[super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT
[12:28:23] Message queue limit is set to 0 which may lead to potential OOMEs
when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to
message queues growth on sender and receiver sides.
[12:28:23] Security status [authentication=off, tls/ssl=off]
[12:28:23] Joining node doesn't have encryption data
[node=3205dca1-2c61-4f76-8475-e0c5c0f1]
[12:28:23] Topology snapshot [ver=2, locNode=a34d6c79, servers=2, clients=0,
state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB]
[12:28:29] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[12:28:29] Data Regions Configured:
[12:28:29] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB,
persistence=false, lazyMemoryAllocation=true]
[12:28:29]
[12:28:29] Ignite node started OK (id=3205dca1, instance
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987950#comment-16987950 ] Maxim Muzafarov commented on IGNITE-11704: -- Commit reverted as discussed on dev-list. The issue reopened. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Priority: Major > Labels: rebalance > Fix For: 2.8 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987788#comment-16987788
]
Maxim Muzafarov commented on IGNITE-11704:
--
Flaky failure with corrupted B+Tree in the master branch.
https://ci.ignite.apache.org/viewLog.html?buildId=4807946=buildResultsDiv=IgniteTests24Java8_Cache9#testNameId1910487508546147692
{code:java}
class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=1544803905,
val2=844420635196573]], msg=Runtime failure on cursor iteration]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:5927)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5438)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next(BPlusTree.java:5661)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl$1.next(IgniteCacheOffheapManagerImpl.java:3020)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition$2.hasNextX(GridDhtLocalPartition.java:1226)
at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.doClear(GridDhtLocalPartition.java:1295)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearTombstones(GridDhtLocalPartition.java:1242)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$ClearTombstonesTask.run0(PartitionsEvictManager.java:673)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$AbstractEvictionTask.run(PartitionsEvictManager.java:587)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7061)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.AssertionError: Key is not ready: CacheDataRowAdapter
[key=null, val=null, expireTime=-1, ver=null, cacheId=0, link=0001003c77d8]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.key(CacheDataRowAdapter.java:837)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.compare(CacheDataTree.java:382)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.compare(CacheDataTree.java:63)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5200)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.findLowerBound(BPlusTree.java:5317)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5588)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5376)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5428)
... 14 more {code}
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Pavel Kovalenko
>Priority: Major
> Labels: rebalance
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953592#comment-16953592 ] Alexei Scherbakov commented on IGNITE-11704: [~jokser] Looks good. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Pavel Kovalenko >Priority: Major > Labels: rebalance > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953552#comment-16953552
]
Ignite TC Bot commented on IGNITE-11704:
{panel:title=Branch: [pull/6931/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=4701586buildTypeId=IgniteTests24Java8_RunAll]
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Pavel Kovalenko
>Priority: Major
> Labels: rebalance
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an incomplete
> baseline).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952020#comment-16952020 ] Alexei Scherbakov commented on IGNITE-11704: [~jokser] 4. Typo in the comment. My understanding is the code will be called when datastreamer initiates first update for an entry, is it true ? 6. * Looks like it's not necessary to preload 256k keys for historical rebalance, you need only one update in each partition. * Test looks similar but my idea is to delay each batch and remove all containing keys in the batch, then release batch. Such scenario should bring all partition keys to tombstones and looks interesting. In other aspects looks good. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Pavel Kovalenko >Priority: Major > Labels: rebalance > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951983#comment-16951983 ] Pavel Kovalenko commented on IGNITE-11704: -- [~ascherbakov] I've fixed several of your concerns others are commented: 1. Tombstone object storing has optimized: Any object (key or value) has a header (object len + object type). The object type can be regular, binary or byte array. In the previous version, the tombstone will be regular cache object with marshalled "null" value. In current version, I introduced a special type of object - Tombstone that doesn't store any value, only header. All checking for the tombstone has optimized. 2. I think it's fine. Every clear tombstone task periodically check is partition become in not OWNING state. In this case, a clear tombstones operation is stopped. Yes, there can be a window of time where both clear tombstones and eviction can happen, but it shouldn't be long. 3. DropCacheContextDuringEvictionTest is reworked due to reuse test PartitionsEvictManagerAbstractTest for checking tomstones failure. cacheGroupMetricsRegistryName is added as a utility method as part of cache group tombstone metrics. GridCommandHandlerIndexingTest - merge artifact, should be ignored. 4. I've added a comment when this condition is true. 5. This test already exists (org.apache.ignite.internal.processors.cache.distributed.CacheRemoveWithTombstonesTest#testRemoveAndRebalanceRaceTx) 6. I've reworked code and now clearAll and clearTombstones have a common codebase. Could you please review it again? > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Pavel Kovalenko >Priority: Major > Labels: rebalance > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948360#comment-16948360 ] Alexei Scherbakov commented on IGNITE-11704: 6. GridDhtLocalPartition.clearTombstones looks very similar to GridDhtLocalPartition.clearAll. Could we avoid code duplication ? > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Pavel Kovalenko >Priority: Major > Labels: rebalance > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946592#comment-16946592 ] Alexei Scherbakov commented on IGNITE-11704: 5. I would add one more load test scenario: Start a node, backups=1. Load many keys (like 100k). Join another node triggering rebalance. Delay each batch. Remove keys supplied in the batch. Release batch. Validate cache is empty and tombstones are cleared. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Pavel Kovalenko >Priority: Major > Labels: rebalance > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946165#comment-16946165
]
Alexei Scherbakov commented on IGNITE-11704:
[~jokser] [~sboikov]
I've reviewed changes. Overall looks good, but still I have some questions.
1. My main concern is regarding the necessity of tombstoneBytes 5-bytes object.
Seems it's possible to implement tombstone by treating absence of value as a
tombstone.
For example, valLen=0 could be treated as tombstone presense. Doing so we can
get rid of 5 bytes comparison, and instead do null check:
{noformat}
private Boolean isTombstone(ByteBuffer buf, int offset) {
int valLen = buf.getInt(buf.position() + offset);
if (valLen != tombstoneBytes.length)
return Boolean.FALSE;
...
}
{noformat}
Instead we can do something like {{if (valLen == 0) return true}}
2. With new changes in PartitionsEvictManager it's possible to have two tasks
of different types for the same partition.
Consider a scenario:
* node finished rebalancing and starts to clear thombstones
* another node joins topology and become an owner for clearing partition.
* eviction is started for already clearing partition.
Probably this should not be allowed.
3. I see changes having no obvious relation to contribution, for example:
static String cacheGroupMetricsRegistryName(String cacheGrp)
DropCacheContextDuringEvictionTest.java
GridCommandHandlerIndexingTest.java
What's the purpose of these ?
4. Could you explain the modification in
org.apache.ignite.internal.processors.cache.GridCacheMapEntry#initialValue:
update0 |= (!preload && val == null); ?
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Pavel Kovalenko
>Priority: Major
> Labels: rebalance
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an incomplete
> baseline).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944348#comment-16944348
]
Ignite TC Bot commented on IGNITE-11704:
{panel:title=Branch: [pull/6931/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=4657057buildTypeId=IgniteTests24Java8_RunAll]
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Pavel Kovalenko
>Priority: Major
> Labels: rebalance
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an incomplete
> baseline).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920882#comment-16920882 ] Pavel Kovalenko commented on IGNITE-11704: -- [~sboikov] If you don't mind I'll take the ticket for finishing. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Pavel Kovalenko >Priority: Major > Labels: rebalance > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900252#comment-16900252 ] Pavel Kovalenko commented on IGNITE-11704: -- [~sboikov] Thank you for contribution. I have a couple of questions and suggestions regarding the change: 1) I think we should remain org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager#partitionIterator with default behaviour (withTombstones=false). It can help to avoid making changes in existing code using partitionIterator previous behavior. 2) Unnecessary brackets in org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java:1715 3) Why double-check in org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java:1723 is needed? 4) Broken javadoc in org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java:5859 The main concern about change is that tombstones can remain in partition forever if partition CASed to OWNING state and immediately shut down. In this case after node return back it can never clear tombstones. I think "tombstoneCreated" flag should be reflected in partition meta information and saved during a checkpoint. The same information should be added to the appropriate WAL delta record. During recovery, we can notice that partition has tombstones and run the cleaning process. Also, this flag is never reset looking to code. > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Priority: Major > Labels: rebalance > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897261#comment-16897261 ] Semen Boikov commented on IGNITE-11704: --- Implemented initial support for tombstones, changes in branch ignite-11704 (tombstones are created for tx cache while rebalance is in progress): * to do not change data storage format tombstones are stored as a regular rows where value is CacheObject containing marshaled null (1 byte array with binary marshaller). Nulls should not be stored in cache, so it always possible to distinguish tombstones and regular values * tombstones are created for cache removes while partition is MOVING, and asynchronously removed when partition becomes OWNING * tombstones cleanup should not consume too much resources, in this regard it is similar to partition eviction, so I reused PartitionsEvictManager for tombstones cleanup tasks * added special cache store iteration mode for tombstone cleanup (RowData.TOMBSTONES) to read as little data as possible for non-tombstone entries * added per-cache metrics - number of tombstone entries Further tasks (going to add separate JIRAs for this): * when persistence is enabled, can write tombstones on incomplete baseline, then include tombstones in rebalance when node returns, and in this case full partition cleanup on joining node won't be needed * for ATOMIC cache 'deferredDelete' is used to prevent race on backups vs concurrent remove and put. So for atomic caches need some other logic to understand when it is safe to remove tombstones > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Semen Boikov >Priority: Major > Labels: rebalance > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897087#comment-16897087
]
Ignite TC Bot commented on IGNITE-11704:
{panel:title=Branch: [ignite-11704] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=4423153buildTypeId=IgniteTests24Java8_RunAll]
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Semen Boikov
>Priority: Major
> Labels: rebalance
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an incomplete
> baseline).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896866#comment-16896866
]
Ignite TC Bot commented on IGNITE-11704:
{panel:title=Branch: [ignite-11704] Base: [master] : Possible Blockers
(3)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Cache 9{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=4432456]]
* IgniteCacheTestSuite9:
CacheRemoveWithTombstonesLoadTest.testRemoveAndRebalanceWithPersistence - New
test duration 79s is more that 1 minute
{color:#d04437}MVCC Cache 9{color} [[tests
2|https://ci.ignite.apache.org/viewLog.html?buildId=4432458]]
* IgniteCacheMvccTestSuite9:
CacheRemoveWithTombstonesLoadTest.testRemoveAndRebalance - New test duration
81s is more that 1 minute
* IgniteCacheMvccTestSuite9:
CacheRemoveWithTombstonesLoadTest.testRemoveAndRebalanceWithPersistence - New
test duration 113s is more that 1 minute
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=4423153buildTypeId=IgniteTests24Java8_RunAll]
> Write tombstones during rebalance to get rid of deferred delete buffer
> --
>
> Key: IGNITE-11704
> URL: https://issues.apache.org/jira/browse/IGNITE-11704
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexey Goncharuk
>Assignee: Semen Boikov
>Priority: Major
> Labels: rebalance
>
> Currently Ignite relies on deferred delete buffer in order to handle
> write-remove conflicts during rebalance. Given the limit size of the buffer,
> this approach is fundamentally flawed, especially in case when persistence is
> enabled.
> I suggest to extend the logic of data storage to be able to store key
> tombstones - to keep version for deleted entries. The tombstones will be
> stored when rebalance is in progress and should be cleaned up when rebalance
> is completed.
> Later this approach may be used to implement fast partition rebalance based
> on merkle trees (in this case, tombstones should be written on an incomplete
> baseline).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[
https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894634#comment-16894634
]
Ignite TC Bot commented on IGNITE-11704:
{panel:title=Branch: [ignite-11704] Base: [master] : Possible Blockers
(54)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Data Structures{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=4303754]]
{color:#d04437}Cache 8{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=4303749]]
* IgniteCacheTestSuite8:
PageEvictionPagesRecyclingAndReusingTest.testPagesRecyclingAndReusingTxReplicated
- Test has low fail rate in base branch 0,0% and is not flaky
{color:#d04437}Examples{color} [[tests
5|https://ci.ignite.apache.org/viewLog.html?buildId=4303690]]
* IgniteExamplesSelfTestSuite:
IgniteModelDistributedInferenceExampleSelfTest.testExample - Test has low fail
rate in base branch 0,0% and is not flaky
* IgniteExamplesSelfTestSuite: XGBoostModelParserExampleSelfTest.testExample -
Test has low fail rate in base branch 0,0% and is not flaky
* IgniteExamplesSelfTestSuite: CacheExamplesSelfTest.testCacheQueueExample -
Test has low fail rate in base branch 0,0% and is not flaky
* IgniteExamplesSelfTestSuite:
TensorFlowDistributedInferenceExampleSelfTest.testExample - Test has low fail
rate in base branch 0,0% and is not flaky
* IgniteExamplesSelfTestSuite:
CacheExamplesMultiNodeSelfTest.testCacheQueueExample - Test has low fail rate
in base branch 0,0% and is not flaky
{color:#d04437}Cache 3{color} [[tests
6|https://ci.ignite.apache.org/viewLog.html?buildId=4303744]]
* IgniteBinaryObjectsCacheTestSuite3:
IgniteCacheGroupsTest.testContinuousQueriesMultipleGroups2 - Test has low fail
rate in base branch 0,0% and is not flaky
* IgniteBinaryObjectsCacheTestSuite3: IgniteCacheGroupsTest.testInterceptors -
Test has low fail rate in base branch 0,0% and is not flaky
* IgniteBinaryObjectsCacheTestSuite3: IgniteCacheGroupsTest.testCacheStores -
Test has low fail rate in base branch 0,0% and is not flaky
* IgniteBinaryObjectsCacheTestSuite3: IgniteCacheGroupsTest.testAffinityMappers
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteBinaryObjectsCacheTestSuite3: IgniteCacheGroupsTest.testStartManyCaches
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteBinaryObjectsCacheTestSuite3:
IgniteCacheGroupsTest.testContinuousQueriesMultipleGroups1 - Test has low fail
rate in base branch 0,0% and is not flaky
{color:#d04437}PDS 2{color} [[tests
2|https://ci.ignite.apache.org/viewLog.html?buildId=4303761]]
* IgnitePdsTestSuite2: IgnitePersistentStoreDataStructuresTest.testSet - Test
has low fail rate in base branch 0,0% and is not flaky
* IgnitePdsTestSuite2: IgnitePersistentStoreDataStructuresTest.testQueue - Test
has low fail rate in base branch 0,0% and is not flaky
{color:#d04437}ZooKeeper (Discovery) 3{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=4303772]]
{color:#d04437}Cache 5{color} [[tests
4|https://ci.ignite.apache.org/viewLog.html?buildId=4303746]]
* IgniteCacheWithIndexingTestSuite: ClusterReadOnlyModeSqlTest.testSqlReadOnly
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCacheTestSuite5: ClusterReadOnlyModeTest.testCacheGetPutRemove - Test
has low fail rate in base branch 0,0% and is not flaky
* IgniteCacheTestSuite5:
IgniteCachePartitionLossPolicySelfTest.testReadWriteSafeWithBackups[ATOMIC] -
Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCacheWithIndexingTestSuite: IgniteCacheGroupsSqlTest.testJoinQuery3 -
Test has low fail rate in base branch 0,0% and is not flaky
{color:#d04437}Cache (Restarts) 1{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=4303739]]
{color:#d04437}Cache 1{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=4303742]]
* IgniteBinaryCacheTestSuite:
CacheDeferredDeleteSanitySelfTest.testDeferredDelete - Test has low fail rate
in base branch 0,0% and is not flaky
{color:#d04437}Cache (Failover) 1{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=4303734]]
* IgniteCacheFailoverTestSuite:
IgniteAtomicLongChangingTopologySelfTest.testClientCollocatedSetCreateCloseFailover
- Test has low fail rate in base branch 0,0% and is not flaky
{color:#d04437}Cache 9{color} [[tests
2|https://ci.ignite.apache.org/viewLog.html?buildId=4303750]]
* IgniteCacheTestSuite9: IoStatisticsCacheSelfTest.testCacheGroupCaches - Test
has low fail rate in base branch 0,0% and is not flaky
* IgniteCacheTestSuite9:
IoStatisticsCachePersistenceSelfTest.testCacheGroupCaches - Test has low fail
rate in base branch 0,0% and is not flaky
{color:#d04437}Service Grid{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=4303776]]
* IgniteServiceGridTestSuite:
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877900#comment-16877900 ] Semen Boikov commented on IGNITE-11704: --- [~Mmuzaf], yes, please move to unassigned. Thank you! > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Maxim Muzafarov >Priority: Major > Labels: rebalance > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877882#comment-16877882 ] Maxim Muzafarov commented on IGNITE-11704: -- [~sboikov], Currently, I've haven't started working on this issue yet. If you want to - I can move it to unassigned. Will you? > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Maxim Muzafarov >Priority: Major > Labels: rebalance > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-11704) Write tombstones during rebalance to get rid of deferred delete buffer
[ https://issues.apache.org/jira/browse/IGNITE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877869#comment-16877869 ] Semen Boikov commented on IGNITE-11704: --- [~Mmuzaf], do you plan to work on this issue? > Write tombstones during rebalance to get rid of deferred delete buffer > -- > > Key: IGNITE-11704 > URL: https://issues.apache.org/jira/browse/IGNITE-11704 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Goncharuk >Assignee: Maxim Muzafarov >Priority: Major > Labels: rebalance > > Currently Ignite relies on deferred delete buffer in order to handle > write-remove conflicts during rebalance. Given the limit size of the buffer, > this approach is fundamentally flawed, especially in case when persistence is > enabled. > I suggest to extend the logic of data storage to be able to store key > tombstones - to keep version for deleted entries. The tombstones will be > stored when rebalance is in progress and should be cleaned up when rebalance > is completed. > Later this approach may be used to implement fast partition rebalance based > on merkle trees (in this case, tombstones should be written on an incomplete > baseline). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
