[jira] [Commented] (IGNITE-6579) WAL history does not used when node returns to cluster again
[
https://issues.apache.org/jira/browse/IGNITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369104#comment-16369104
]
Pavel Pereslegin commented on IGNITE-6579:
--
[~v.pyatkov],
I think it's better to create a separate ticket to improve the logging and
close this one (as Won't FIX).
> WAL history does not used when node returns to cluster again
>
>
> Key: IGNITE-6579
> URL: https://issues.apache.org/jira/browse/IGNITE-6579
> Project: Ignite
> Issue Type: Bug
> Components: persistence
>Reporter: Vladislav Pyatkov
>Priority: Major
>
> When I have set big enough value to "WAL history size" and stop node on 20
> minutes, I got the message from coordinator (order=1):
> {noformat}
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2424,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2427,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2426,
> haveHistory=false]
> {noformat}
> after start node again.
> I think, history size should be enough, but I see it is not by logs
> (haveHistory=false).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-6579) WAL history does not used when node returns to cluster again
[
https://issues.apache.org/jira/browse/IGNITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328816#comment-16328816
]
Pavel Pereslegin commented on IGNITE-6579:
--
Hello, [~v.pyatkov].
I agreed, but since coordinator doesn't have information about non local
partition sizes we can't simply point to this restriction here, at least
without some irrelevant changes to existing partition exchange procedure.
To improve understanding, may be, we can add debug message on nodes where WAL
history is being reserved for exchange (in case partition size too small for
partial rebalancing)?
> WAL history does not used when node returns to cluster again
>
>
> Key: IGNITE-6579
> URL: https://issues.apache.org/jira/browse/IGNITE-6579
> Project: Ignite
> Issue Type: Bug
> Components: persistence
>Reporter: Vladislav Pyatkov
>Priority: Major
>
> When I have set big enough value to "WAL history size" and stop node on 20
> minutes, I got the message from coordinator (order=1):
> {noformat}
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2424,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2427,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2426,
> haveHistory=false]
> {noformat}
> after start node again.
> I think, history size should be enough, but I see it is not by logs
> (haveHistory=false).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-6579) WAL history does not used when node returns to cluster again
[
https://issues.apache.org/jira/browse/IGNITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303257#comment-16303257
]
Vladislav Pyatkov commented on IGNITE-6579:
---
[~xtern]
If you are right, you should to show correct message.
Instead of {{haveHistory=false}}, would by better to see something like:
{{Size of partition too little, full rebalanse will be passed.}}
> WAL history does not used when node returns to cluster again
>
>
> Key: IGNITE-6579
> URL: https://issues.apache.org/jira/browse/IGNITE-6579
> Project: Ignite
> Issue Type: Bug
> Components: persistence
>Reporter: Vladislav Pyatkov
>
> When I have set big enough value to "WAL history size" and stop node on 20
> minutes, I got the message from coordinator (order=1):
> {noformat}
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2424,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2427,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2426,
> haveHistory=false]
> {noformat}
> after start node again.
> I think, history size should be enough, but I see it is not by logs
> (haveHistory=false).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (IGNITE-6579) WAL history does not used when node returns to cluster again
[
https://issues.apache.org/jira/browse/IGNITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303251#comment-16303251
]
Pavel Pereslegin commented on IGNITE-6579:
--
Hello [~v.pyatkov],
I tried the following scenario:
# Set JVM option -DIGNITE_PDS_WAL_REBALANCE_THRESHOLD=1.
# Start nodes A, B and C with one replicated cache (no backups,
RendezvousAffinityFunction with 10 partitions).
# Put 10 values to cache (1 keys per partition).
# Stop node C.
# Put 3000 values to cache (10300 keys per partition).
# Rejoin node C (nodeId = 606c6f4d-c314-4345-8b6d-cc3f3792).
# Observing messages from coordinator (haveHistory=true).
{noformat}
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=0, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=1, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=2, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=3, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=4, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=5, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=6, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=7, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=8, haveHistory=true]
[GridDhtPartitionTopologyImpl] Partition has been scheduled for rebalancing due
to outdated update counter [nodeId=606c6f4d-c314-4345-8b6d-cc3f3792,
cacheOrGroupName=default, partId=9, haveHistory=true]
{noformat}
If I set IGNITE_PDS_WAL_REBALANCE_THRESHOLD larger than the partition size
(10301 for example) - WAL history is not used.
{noformat}
Partition has been scheduled for rebalancing due to outdated update counter
[nodeId=606c6f4d-c314-4345-8b6d-cc3f3792, cacheOrGroupName=default,
partId=0, haveHistory=false]
Partition has been scheduled for rebalancing due to outdated update counter
[nodeId=606c6f4d-c314-4345-8b6d-cc3f3792, cacheOrGroupName=default,
partId=1, haveHistory=false]
...{noformat}
> WAL history does not used when node returns to cluster again
>
>
> Key: IGNITE-6579
> URL: https://issues.apache.org/jira/browse/IGNITE-6579
> Project: Ignite
> Issue Type: Bug
> Components: persistence
>Reporter: Vladislav Pyatkov
>
> When I have set big enough value to "WAL history size" and stop node on 20
> minutes, I got the message from coordinator (order=1):
> {noformat}
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2424,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2427,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2426,
> haveHistory=false]
> {noformat}
> after start node again.
> I think, history size should be enough, but I see it is not by logs
>
[jira] [Commented] (IGNITE-6579) WAL history does not used when node returns to cluster again
[
https://issues.apache.org/jira/browse/IGNITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303139#comment-16303139
]
Vladislav Pyatkov commented on IGNITE-6579:
---
[~xtern]
Setting IGNITE_PDS_WAL_REBALANCE_THRESHOLD to 0 is very extreme condition.
Please, try to set it more then zero, and purposely puts know quantity of keys
to partition (in order to the quantity will be more then the
REBALANCE_THRESHOLD).
> WAL history does not used when node returns to cluster again
>
>
> Key: IGNITE-6579
> URL: https://issues.apache.org/jira/browse/IGNITE-6579
> Project: Ignite
> Issue Type: Bug
> Components: persistence
>Reporter: Vladislav Pyatkov
>
> When I have set big enough value to "WAL history size" and stop node on 20
> minutes, I got the message from coordinator (order=1):
> {noformat}
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2424,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2427,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2426,
> haveHistory=false]
> {noformat}
> after start node again.
> I think, history size should be enough, but I see it is not by logs
> (haveHistory=false).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (IGNITE-6579) WAL history does not used when node returns to cluster again
[
https://issues.apache.org/jira/browse/IGNITE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301632#comment-16301632
]
Pavel Pereslegin commented on IGNITE-6579:
--
Hello [~v.pyatkov].
Could you provide more details about your configuration and behavior that you
expect to see (or how to reproduce this bug).
I tried following scenario and I believe it works well.
# Set IGNITE_PDS_WAL_REBALANCE_THRESHOLD to 0 (default threshold = 50).
# Start nodes A, B and C with PDS enabled (1 backup).
# Add data to cache.
# Stop node C.
# Add more data to cache.
# Rejoin node C to cluster (nodeId = 93206ec4-0b0a-4c91-bb55-79b05f12).
# Observing messages from coordinator about outdated update counters
(haveHistory=true):
{noformat}
Partition has been scheduled for rebalancing due to outdated update counter
[nodeId=93206ec4-0b0a-4c91-bb55-79b05f12, cacheOrGroupName=default,
partId=0, haveHistory=true]
Partition has been scheduled for rebalancing due to outdated update counter
[nodeId=93206ec4-0b0a-4c91-bb55-79b05f12, cacheOrGroupName=default,
partId=1, haveHistory=true]
Partition has been scheduled for rebalancing due to outdated update counter
[nodeId=93206ec4-0b0a-4c91-bb55-79b05f12, cacheOrGroupName=default,
partId=4, haveHistory=true]
...
{noformat}
> WAL history does not used when node returns to cluster again
>
>
> Key: IGNITE-6579
> URL: https://issues.apache.org/jira/browse/IGNITE-6579
> Project: Ignite
> Issue Type: Bug
> Components: persistence
>Reporter: Vladislav Pyatkov
>
> When I have set big enough value to "WAL history size" and stop node on 20
> minutes, I got the message from coordinator (order=1):
> {noformat}
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2424,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2427,
> haveHistory=false]
> 2017-10-06 15:46:33.429 [WARN
> ][sys-#10740%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.GridDhtPartitionTopologyImpl]
> Partition has been scheduled for rebalancing due to outdated update counter
> [nodeId=e51a1db2-f49b-44a9-b122-adde4016d9e7,
> cacheOrGroupName=CACHEGROUP_PARTICLE_DServiceZone, partId=2426,
> haveHistory=false]
> {noformat}
> after start node again.
> I think, history size should be enough, but I see it is not by logs
> (haveHistory=false).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
