Can someone please help me on this? Thanks, Akash
On Tue, Sep 17, 2019 at 10:40 AM Akash Shinde <[email protected]> wrote: > Hi, > I am trying to recover lost data in case of partition loss. > In my ignite configuration native persistence is *off*. > I have started event listener on EVT_CACHE_REBALANCE_PART_DATA_LOST > event. This listener will get lost partition list > using cache.lostPartitions() method. > The issue is that listener gets call per partition. So if there 100 > partition loss due to single node termination then 100 time this > listener will get called and last multiple calls to the listener will fetch > all lost partition list. > > *Lets take a scenario:* > Started two server nodes Node A and Node B. Started cache with > partition mode and the number of backup set to 0 in order to facilitate > simulation of partition loss scenarios > Started event listener on both node listening to > EVT_CACHE_REBALANCE_PART_DATA_LOST event. > > Number of partitions on node A = 500 > Number of partitions on node B = 524 > > Now stop node B. After termination of node B listener running on node A > gets call multiple time per partition. > I have printed logs on listener > > primary partition size after loss:1024 > *Lost partion Nos.1* > IgniteThread [compositeRwLockIdx=1, stripe=-1, plc=-1, > name=exchange-worker-#42%springDataNode%]::*[0]* > Event Detail:CacheRebalancingEvent [cacheName=ASSET_GROUP_CACHE, part=0, > discoNode=TcpDiscoveryNode [id=1bb17828-3556-499f-a4e6-98cfdc1d11fb, > addrs=[0:0:0:0:0:0:0:1, 10.113.14.98, 127.0.0.1], sockAddrs=[], > discPort=47501, order=2, intOrder=2, lastExchangeTime=1568357181089, > loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], > discoEvtType=12, discoTs=1568357376683, discoEvtName=NODE_FAILED, > nodeId8=499400ac, msg=Cache rebalancing event., > type=CACHE_REBALANCE_PART_DATA_LOST, tstamp=1568357376714] > primary partition size after loss:1024 > *Lost partion Nos.2* > IgniteThread [compositeRwLockIdx=1, stripe=-1, plc=-1, > name=exchange-worker-#42%springDataNode%]::*[0, 1]* > Event Detail:CacheRebalancingEvent [cacheName=ASSET_GROUP_CACHE, part=1, > discoNode=TcpDiscoveryNode [id=1bb17828-3556-499f-a4e6-98cfdc1d11fb, > addrs=[0:0:0:0:0:0:0:1, 10.113.14.98, 127.0.0.1], sockAddrs=[], > discPort=47501, order=2, intOrder=2, lastExchangeTime=1568357181089, > loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], > discoEvtType=12, discoTs=1568357376683, discoEvtName=NODE_FAILED, > nodeId8=499400ac, msg=Cache rebalancing event., > type=CACHE_REBALANCE_PART_DATA_LOST, tstamp=1568357376726] > primary partition size after loss:1024 > *Lost partion Nos.3* > IgniteThread [compositeRwLockIdx=1, stripe=-1, plc=-1, > name=exchange-worker-#42%springDataNode%]::*[0, 1, 2]* > Event Detail:CacheRebalancingEvent [cacheName=ASSET_GROUP_CACHE, part=2, > discoNode=TcpDiscoveryNode [id=1bb17828-3556-499f-a4e6-98cfdc1d11fb, > addrs=[0:0:0:0:0:0:0:1, 10.113.14.98, 127.0.0.1], sockAddrs=[], > discPort=47501, order=2, intOrder=2, lastExchangeTime=1568357181089, > loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], > discoEvtType=12, discoTs=1568357376683, discoEvtName=NODE_FAILED, > nodeId8=499400ac, msg=Cache rebalancing event., > type=CACHE_REBALANCE_PART_DATA_LOST, tstamp=1568357376726] > primary partition size after loss:1024 > *Lost partion Nos.4* > IgniteThread [compositeRwLockIdx=1, stripe=-1, plc=-1, > name=exchange-worker-#42%springDataNode%]::*[0, 1, 2, 4]* > Event Detail:CacheRebalancingEvent [cacheName=ASSET_GROUP_CACHE, part=4, > discoNode=TcpDiscoveryNode [id=1bb17828-3556-499f-a4e6-98cfdc1d11fb, > addrs=[0:0:0:0:0:0:0:1, 10.113.14.98, 127.0.0.1], sockAddrs=[], > discPort=47501, order=2, intOrder=2, lastExchangeTime=1568357181089, > loc=false, ver=2.6.0#20180710-sha1:669feacc, isClient=false], > discoEvtType=12, discoTs=1568357376683, discoEvtName=NODE_FAILED, > nodeId8=499400ac, msg=Cache rebalancing event., > type=CACHE_REBALANCE_PART_DATA_LOST, tstamp=1568357376736] > primary partition size after loss:1024 > *Lost partion Nos.5* > *.* > *.* > *.* > *.* > IgniteThread [compositeRwLockIdx=1, stripe=-1, plc=-1, > name=exchange-worker-#42%springDataNode%]::[0, 1, 2, 4, 5, 6, 7, 11, 13, > 17, 22, 26, 28, 29, 30, 33, 34, 37, 38, 41, 43, 45, 47, 48, 49, 50, 55, 58, > 61, 62, 64, 65, 68, 70, 71, 75, 77, 79, 81, 82, 85, 87, 88, 89, 90, 93, > 100, 101, 102, 104, 110, 112, 114, 116, 121, 123, 125, 126, 132, 133, 135, > 137, 138, 139, 140, 144, 145, 146, 147, 149, 150, 151, 154, 156, 157, 158, > 163, 164, 165, 169, 170, 172, 173, 176, 178, 180, 182, 183, 184, 185, 195, > 196, 198, 199, 203, 204, 212, 213, 215, 217, 219, 220, 222, 223, 224, 226, > 227, 230, 233, 234, 236, 237, 240, 242, 245, 248, 250, 251, 253, 255, 257, > 258, 263, 265, 266, 267, 269, 270, 272, 273, 275, 276, 277, 278, 281, 282, > 283, 287, 288, 292, 293, 295, 296, 297, 298, 300, 301, 302, 305, 308, 309, > 310, 311, 313, 314, 315, 318, 319, 320, 322, 323, 324, 326, 327, 328, 329, > 330, 331, 332, 333, 336, 340, 342, 344, 347, 348, 349, 351, 352, 353, 354, > 355, 357, 362, 364, 369, 370, 371, 373, 374, 375, 376, 380, 382, 383, 387, > 389, 394, 395, 396, 397, 398, 401, 402, 403, 407, 408, 409, 410, 411, 412, > 413, 416, 417, 421, 424, 425, 427, 430, 431, 433, 435, 437, 438, 439, 440, > 441, 442, 443, 445, 446, 452, 454, 455, 456, 459, 461, 463, 466, 470, 472, > 474, 475, 476, 480, 481, 482, 484, 485, 489, 492, 494, 495, 496, 497, 498, > 499, 501, 502, 503, 504, 505, 508, 510, 511, 512, 513, 514, 515, 516, 519, > 523, 525, 526, 527, 529, 530, 531, 532, 535, 536, 539, 540, 541, 543, 545, > 546, 550, 552, 553, 555, 557, 560, 569, 572, 573, 575, 576, 579, 582, 589, > 591, 593, 594, 597, 599, 602, 603, 604, 605, 607, 608, 610, 612, 613, 614, > 615, 616, 617, 619, 622, 624, 625, 626, 627, 630, 631, 632, 633, 634, 635, > 636, 637, 638, 639, 640, 641, 642, 643, 645, 646, 647, 648, 649, 652, 653, > 654, 656, 657, 660, 662, 663, 666, 668, 669, 670, 671, 679, 681, 683, 686, > 688, 691, 693, 698, 701, 702, 703, 705, 706, 709, 712, 713, 716, 717, 719, > 721, 723, 726, 730, 737, 738, 740, 741, 742, 745, 747, 750, 752, 755, 756, > 759, 760, 761, 763, 764, 765, 766, 767, 768, 770, 771, 772, 777, 779, 785, > 786, 789, 790, 792, 793, 794, 799, 801, 804, 811, 816, 818, 822, 823, 824, > 825, 826, 827, 832, 833, 836, 838, 840, 841, 843, 844, 846, 850, 851, 852, > 853, 855, 856, 858, 862, 864, 867, 872, 873, 876, 877, 878, 879, 883, 884, > 886, 887, 890, 892, 895, 897, 898, 899, 900, 902, 903, 904, 905, 906, 907, > 908, 910, 914, 916, 918, 919, 920, 921, 922, 925, 926, 928, 929, 933, 935, > 936, 939, 940, 943, 945, 950, 951, 952, 953, 960, 961, 963, 964, 966, 967, > 972, 973, 975, 977, 979, 980, 982, 983, 984, 985, 987, 989, 991, 992, 995, > 996, 999, 1002, 1003, 1005, 1007, 1011, 1014, 1015, 1016, 1018, 1020, 1021] > Event Detail:CacheRebalancingEvent [cacheName=ASSET_GROUP_CACHE, part= > *412*, discoNode=TcpDiscoveryNode > [id=1bb17828-3556-499f-a4e6-98cfdc1d11fb, addrs=[0:0:0:0:0:0:0:1, > 10.113.14.98, 127.0.0.1], sockAddrs=[], discPort=47501, order=2, > intOrder=2, lastExchangeTime=1568357181089, loc=false, > ver=2.6.0#20180710-sha1:669feacc, isClient=false], discoEvtType=12, > discoTs=1568357376683, discoEvtName=NODE_FAILED, nodeId8=499400ac, > msg=Cache rebalancing event., type=CACHE_REBALANCE_PART_DATA_LOST, > tstamp=1568357423500] > primary partition size after loss:1024 > > *Lost partion Nos.524* > IgniteThread [compositeRwLockIdx=1, stripe=-1, plc=-1, > name=exchange-worker-#42%springDataNode%]::[0, 1, 2, 4, 5, 6, 7, 11, 13, > 17, 22, 26, 28, 29, 30, 33, 34, 37, 38, 41, 43, 45, 47, 48, 49, 50, 55, 58, > 61, 62, 64, 65, 68, 70, 71, 75, 77, 79, 81, 82, 85, 87, 88, 89, 90, 93, > 100, 101, 102, 104, 110, 112, 114, 116, 121, 123, 125, 126, 132, 133, 135, > 137, 138, 139, 140, 144, 145, 146, 147, 149, 150, 151, 154, 156, 157, 158, > 163, 164, 165, 169, 170, 172, 173, 176, 178, 180, 182, 183, 184, 185, 195, > 196, 198, 199, 203, 204, 212, 213, 215, 217, 219, 220, 222, 223, 224, 226, > 227, 230, 233, 234, 236, 237, 240, 242, 245, 248, 250, 251, 253, 255, 257, > 258, 263, 265, 266, 267, 269, 270, 272, 273, 275, 276, 277, 278, 281, 282, > 283, 287, 288, 292, 293, 295, 296, 297, 298, 300, 301, 302, 305, 308, 309, > 310, 311, 313, 314, 315, 318, 319, 320, 322, 323, 324, 326, 327, 328, 329, > 330, 331, 332, 333, 336, 340, 342, 344, 347, 348, 349, 351, 352, 353, 354, > 355, 357, 362, 364, 369, 370, 371, 373, 374, 375, 376, 380, 382, 383, 387, > 389, 394, 395, 396, 397, 398, 401, 402, 403, 407, 408, 409, 410, 411, 412, > 413, 416, 417, 421, 424, 425, 427, 430, 431, 433, 435, 437, 438, 439, 440, > 441, 442, 443, 445, 446, 452, 454, 455, 456, 459, 461, 463, 466, 470, 472, > 474, 475, 476, 480, 481, 482, 484, 485, 489, 492, 494, 495, 496, 497, 498, > 499, 501, 502, 503, 504, 505, 508, 510, 511, 512, 513, 514, 515, 516, 519, > 523, 525, 526, 527, 529, 530, 531, 532, 535, 536, 539, 540, 541, 543, 545, > 546, 550, 552, 553, 555, 557, 560, 569, 572, 573, 575, 576, 579, 582, 589, > 591, 593, 594, 597, 599, 602, 603, 604, 605, 607, 608, 610, 612, 613, 614, > 615, 616, 617, 619, 622, 624, 625, 626, 627, 630, 631, 632, 633, 634, 635, > 636, 637, 638, 639, 640, 641, 642, 643, 645, 646, 647, 648, 649, 652, 653, > 654, 656, 657, 660, 662, 663, 666, 668, 669, 670, 671, 679, 681, 683, 686, > 688, 691, 693, 698, 701, 702, 703, 705, 706, 709, 712, 713, 716, 717, 719, > 721, 723, 726, 730, 737, 738, 740, 741, 742, 745, 747, 750, 752, 755, 756, > 759, 760, 761, 763, 764, 765, 766, 767, 768, 770, 771, 772, 777, 779, 785, > 786, 789, 790, 792, 793, 794, 799, 801, 804, 811, 816, 818, 822, 823, 824, > 825, 826, 827, 832, 833, 836, 838, 840, 841, 843, 844, 846, 850, 851, 852, > 853, 855, 856, 858, 862, 864, 867, 872, 873, 876, 877, 878, 879, 883, 884, > 886, 887, 890, 892, 895, 897, 898, 899, 900, 902, 903, 904, 905, 906, 907, > 908, 910, 914, 916, 918, 919, 920, 921, 922, 925, 926, 928, 929, 933, 935, > 936, 939, 940, 943, 945, 950, 951, 952, 953, 960, 961, 963, 964, 966, 967, > 972, 973, 975, 977, 979, 980, 982, 983, 984, 985, 987, 989, 991, 992, 995, > 996, 999, 1002, 1003, 1005, 1007, 1011, 1014, 1015, 1016, 1018, 1020, 1021] > Event Detail:CacheRebalancingEvent [cacheName=ASSET_GROUP_CACHE, part= > *413*, discoNode=TcpDiscoveryNode > [id=1bb17828-3556-499f-a4e6-98cfdc1d11fb, addrs=[0:0:0:0:0:0:0:1, > 10.113.14.98, 127.0.0.1], sockAddrs=[], discPort=47501, order=2, > intOrder=2, lastExchangeTime=1568357181089, loc=false, > ver=2.6.0#20180710-sha1:669feacc, isClient=false], discoEvtType=12, > discoTs=1568357376683, discoEvtName=NODE_FAILED, nodeId8=499400ac, > msg=Cache rebalancing event., type=CACHE_REBALANCE_PART_DATA_LOST, > tstamp=1568357423500] > primary partition size after loss:1024 > > *Lost partion No.524* > > *The number of lost partitions gets increment on each consecutive event > call and the last many calls to listener have complete list of lost > partitions.* > *Questions: 1)Is there any way get list of complete lost partitions? > Because I want start cache loading for these partitions. Its getting > difficult to determine when to call cache loading due event call on per > partition.* > * 2)I want reset only partitions using > *resetLostPartitions() > *those I have handled in partition lost event listener. * > > Thanks, > Akash >
