(send this mail again to remove "Spam Mail" on title)
Dear steve:
I add a printf in the following code.
And I got many printf when the problem happen.
I don't understand what is throw away.
Is there any doc or explanation I could reference?
Thanks.
Jay...
====== code
if (throw_away_mode == THROW_AWAY_ACTIVE) {
printf("THROW_AWAY_ACTIVE mcast->fragmented %d\n",
mcast->fragmented);
/* Throw away the first msg block */
if (mcast->fragmented == 0 || mcast->fragmented == 1) {
throw_away_mode = THROW_AWAY_INACTIVE;
assembly->index += msg_lens[0];
iov_delv.iov_base =
&assembly->data[assembly->index];
====== debug output
sync_start_process
sync_start_process call sync_barrier_send with res 0
THROW_AWAY_ACTIVE mcast->fragmented 150
THROW_AWAY_ACTIVE mcast->fragmented 151
THROW_AWAY_ACTIVE mcast->fragmented 152
THROW_AWAY_ACTIVE mcast->fragmented 153
THROW_AWAY_ACTIVE mcast->fragmented 154
THROW_AWAY_ACTIVE mcast->fragmented 155
THROW_AWAY_ACTIVE mcast->fragmented 156
THROW_AWAY_ACTIVE mcast->fragmented 157
THROW_AWAY_ACTIVE mcast->fragmented 158
THROW_AWAY_ACTIVE mcast->fragmented 159
THROW_AWAY_ACTIVE mcast->fragmented 160
THROW_AWAY_ACTIVE mcast->fragmented 161
THROW_AWAY_ACTIVE mcast->fragmented 162
THROW_AWAY_ACTIVE mcast->fragmented 163
THROW_AWAY_ACTIVE mcast->fragmented 164
THROW_AWAY_ACTIVE mcast->fragmented 165
THROW_AWAY_ACTIVE mcast->fragmented 1
THROW_AWAY_ACTIVE mcast->fragmented 3
THROW_AWAY_ACTIVE mcast->fragmented 4
THROW_AWAY_ACTIVE mcast->fragmented 5
THROW_AWAY_ACTIVE mcast->fragmented 6
THROW_AWAY_ACTIVE mcast->fragmented 7
THROW_AWAY_ACTIVE mcast->fragmented 8
THROW_AWAY_ACTIVE mcast->fragmented 9
THROW_AWAY_ACTIVE mcast->fragmented 10
THROW_AWAY_ACTIVE mcast->fragmented 11
THROW_AWAY_ACTIVE mcast->fragmented 12
THROW_AWAY_ACTIVE mcast->fragmented 13
THROW_AWAY_ACTIVE mcast->fragmented 14
THROW_AWAY_ACTIVE mcast->fragmented 15
THROW_AWAY_ACTIVE mcast->fragmented 16
THROW_AWAY_ACTIVE mcast->fragmented 17
THROW_AWAY_ACTIVE mcast->fragmented 166
THROW_AWAY_ACTIVE mcast->fragmented 167
THROW_AWAY_ACTIVE mcast->fragmented 168
THROW_AWAY_ACTIVE mcast->fragmented 169
THROW_AWAY_ACTIVE mcast->fragmented 170
THROW_AWAY_ACTIVE mcast->fragmented 171
THROW_AWAY_ACTIVE mcast->fragmented 172
THROW_AWAY_ACTIVE mcast->fragmented 173
THROW_AWAY_ACTIVE mcast->fragmented 174
THROW_AWAY_ACTIVE mcast->fragmented 175
THROW_AWAY_ACTIVE mcast->fragmented 176
THROW_AWAY_ACTIVE mcast->fragmented 177
THROW_AWAY_ACTIVE mcast->fragmented 178
THROW_AWAY_ACTIVE mcast->fragmented 179
THROW_AWAY_ACTIVE mcast->fragmented 180
THROW_AWAY_ACTIVE mcast->fragmented 181
THROW_AWAY_ACTIVE mcast->fragmented 18
THROW_AWAY_ACTIVE mcast->fragmented 19
THROW_AWAY_ACTIVE mcast->fragmented 20
THROW_AWAY_ACTIVE mcast->fragmented 21
THROW_AWAY_ACTIVE mcast->fragmented 22
THROW_AWAY_ACTIVE mcast->fragmented 23
THROW_AWAY_ACTIVE mcast->fragmented 24
THROW_AWAY_ACTIVE mcast->fragmented 25
THROW_AWAY_ACTIVE mcast->fragmented 26
THROW_AWAY_ACTIVE mcast->fragmented 27
THROW_AWAY_ACTIVE mcast->fragmented 182
THROW_AWAY_ACTIVE mcast->fragmented 183
THROW_AWAY_ACTIVE mcast->fragmented 184
THROW_AWAY_ACTIVE mcast->fragmented 185
THROW_AWAY_ACTIVE mcast->fragmented 186
THROW_AWAY_ACTIVE mcast->fragmented 187
THROW_AWAY_ACTIVE mcast->fragmented 188
THROW_AWAY_ACTIVE mcast->fragmented 189
THROW_AWAY_ACTIVE mcast->fragmented 190
THROW_AWAY_ACTIVE mcast->fragmented 191
THROW_AWAY_ACTIVE mcast->fragmented 192
THROW_AWAY_ACTIVE mcast->fragmented 193
THROW_AWAY_ACTIVE mcast->fragmented 194
THROW_AWAY_ACTIVE mcast->fragmented 195
THROW_AWAY_ACTIVE mcast->fragmented 196
THROW_AWAY_ACTIVE mcast->fragmented 197
THROW_AWAY_ACTIVE mcast->fragmented 198
THROW_AWAY_ACTIVE mcast->fragmented 28
THROW_AWAY_ACTIVE mcast->fragmented 29
THROW_AWAY_ACTIVE mcast->fragmented 30
THROW_AWAY_ACTIVE mcast->fragmented 31
THROW_AWAY_ACTIVE mcast->fragmented 32
THROW_AWAY_ACTIVE mcast->fragmented 33
THROW_AWAY_ACTIVE mcast->fragmented 34
THROW_AWAY_ACTIVE mcast->fragmented 35
THROW_AWAY_ACTIVE mcast->fragmented 36
THROW_AWAY_ACTIVE mcast->fragmented 199
THROW_AWAY_ACTIVE mcast->fragmented 37
THROW_AWAY_ACTIVE mcast->fragmented 38
THROW_AWAY_ACTIVE mcast->fragmented 39
THROW_AWAY_ACTIVE mcast->fragmented 40
THROW_AWAY_ACTIVE mcast->fragmented 200
THROW_AWAY_ACTIVE mcast->fragmented 201
THROW_AWAY_ACTIVE mcast->fragmented 202
THROW_AWAY_ACTIVE mcast->fragmented 203
THROW_AWAY_ACTIVE mcast->fragmented 204
THROW_AWAY_ACTIVE mcast->fragmented 205
THROW_AWAY_ACTIVE mcast->fragmented 206
THROW_AWAY_ACTIVE mcast->fragmented 207
THROW_AWAY_ACTIVE mcast->fragmented 208
THROW_AWAY_ACTIVE mcast->fragmented 209
THROW_AWAY_ACTIVE mcast->fragmented 210
THROW_AWAY_ACTIVE mcast->fragmented 211
THROW_AWAY_ACTIVE mcast->fragmented 212
THROW_AWAY_ACTIVE mcast->fragmented 213
THROW_AWAY_ACTIVE mcast->fragmented 214
THROW_AWAY_ACTIVE mcast->fragmented 215
THROW_AWAY_ACTIVE mcast->fragmented 216
THROW_AWAY_ACTIVE mcast->fragmented 1
THROW_AWAY_ACTIVE mcast->fragmented 3
THROW_AWAY_ACTIVE mcast->fragmented 4
THROW_AWAY_ACTIVE mcast->fragmented 5
THROW_AWAY_ACTIVE mcast->fragmented 6
THROW_AWAY_ACTIVE mcast->fragmented 217
THROW_AWAY_ACTIVE mcast->fragmented 218
THROW_AWAY_ACTIVE mcast->fragmented 219
THROW_AWAY_ACTIVE mcast->fragmented 220
THROW_AWAY_ACTIVE mcast->fragmented 221
THROW_AWAY_ACTIVE mcast->fragmented 222
THROW_AWAY_ACTIVE mcast->fragmented 223
THROW_AWAY_ACTIVE mcast->fragmented 224
THROW_AWAY_ACTIVE mcast->fragmented 225
THROW_AWAY_ACTIVE mcast->fragmented 226
THROW_AWAY_ACTIVE mcast->fragmented 227
THROW_AWAY_ACTIVE mcast->fragmented 228
THROW_AWAY_ACTIVE mcast->fragmented 229
THROW_AWAY_ACTIVE mcast->fragmented 230
THROW_AWAY_ACTIVE mcast->fragmented 231
THROW_AWAY_ACTIVE mcast->fragmented 232
THROW_AWAY_ACTIVE mcast->fragmented 233
THROW_AWAY_ACTIVE mcast->fragmented 7
THROW_AWAY_ACTIVE mcast->fragmented 8
THROW_AWAY_ACTIVE mcast->fragmented 9
THROW_AWAY_ACTIVE mcast->fragmented 234
THROW_AWAY_ACTIVE mcast->fragmented 235
THROW_AWAY_ACTIVE mcast->fragmented 236
THROW_AWAY_ACTIVE mcast->fragmented 237
THROW_AWAY_ACTIVE mcast->fragmented 238
THROW_AWAY_ACTIVE mcast->fragmented 239
THROW_AWAY_ACTIVE mcast->fragmented 240
THROW_AWAY_ACTIVE mcast->fragmented 241
THROW_AWAY_ACTIVE mcast->fragmented 242
THROW_AWAY_ACTIVE mcast->fragmented 243
THROW_AWAY_ACTIVE mcast->fragmented 244
THROW_AWAY_ACTIVE mcast->fragmented 245
THROW_AWAY_ACTIVE mcast->fragmented 246
THROW_AWAY_ACTIVE mcast->fragmented 247
THROW_AWAY_ACTIVE mcast->fragmented 248
THROW_AWAY_ACTIVE mcast->fragmented 249
THROW_AWAY_ACTIVE mcast->fragmented 250
THROW_AWAY_ACTIVE mcast->fragmented 10
THROW_AWAY_ACTIVE mcast->fragmented 11
THROW_AWAY_ACTIVE mcast->fragmented 12
THROW_AWAY_ACTIVE mcast->fragmented 13
THROW_AWAY_ACTIVE mcast->fragmented 251
THROW_AWAY_ACTIVE mcast->fragmented 252
THROW_AWAY_ACTIVE mcast->fragmented 253
THROW_AWAY_ACTIVE mcast->fragmented 254
THROW_AWAY_ACTIVE mcast->fragmented 255
THROW_AWAY_ACTIVE mcast->fragmented 1
THROW_AWAY_ACTIVE mcast->fragmented 3
THROW_AWAY_ACTIVE mcast->fragmented 4
THROW_AWAY_ACTIVE mcast->fragmented 5
THROW_AWAY_ACTIVE mcast->fragmented 6
THROW_AWAY_ACTIVE mcast->fragmented 7
THROW_AWAY_ACTIVE mcast->fragmented 8
THROW_AWAY_ACTIVE mcast->fragmented 9
THROW_AWAY_ACTIVE mcast->fragmented 10
THROW_AWAY_ACTIVE mcast->fragmented 11
THROW_AWAY_ACTIVE mcast->fragmented 12
THROW_AWAY_ACTIVE mcast->fragmented 14
THROW_AWAY_ACTIVE mcast->fragmented 13
THROW_AWAY_ACTIVE mcast->fragmented 14
THROW_AWAY_ACTIVE mcast->fragmented 15
THROW_AWAY_ACTIVE mcast->fragmented 16
THROW_AWAY_ACTIVE mcast->fragmented 17
THROW_AWAY_ACTIVE mcast->fragmented 18
THROW_AWAY_ACTIVE mcast->fragmented 19
THROW_AWAY_ACTIVE mcast->fragmented 20
THROW_AWAY_ACTIVE mcast->fragmented 21
THROW_AWAY_ACTIVE mcast->fragmented 22
THROW_AWAY_ACTIVE mcast->fragmented 23
THROW_AWAY_ACTIVE mcast->fragmented 24
THROW_AWAY_ACTIVE mcast->fragmented 25
THROW_AWAY_ACTIVE mcast->fragmented 26
THROW_AWAY_ACTIVE mcast->fragmented 27
THROW_AWAY_ACTIVE mcast->fragmented 28
THROW_AWAY_ACTIVE mcast->fragmented 29
THROW_AWAY_ACTIVE mcast->fragmented 15
THROW_AWAY_ACTIVE mcast->fragmented 16
THROW_AWAY_ACTIVE mcast->fragmented 17
THROW_AWAY_ACTIVE mcast->fragmented 30
THROW_AWAY_ACTIVE mcast->fragmented 31
THROW_AWAY_ACTIVE mcast->fragmented 32
THROW_AWAY_ACTIVE mcast->fragmented 33
THROW_AWAY_ACTIVE mcast->fragmented 34
THROW_AWAY_ACTIVE mcast->fragmented 35
THROW_AWAY_ACTIVE mcast->fragmented 36
THROW_AWAY_ACTIVE mcast->fragmented 37
THROW_AWAY_ACTIVE mcast->fragmented 38
THROW_AWAY_ACTIVE mcast->fragmented 39
THROW_AWAY_ACTIVE mcast->fragmented 40
THROW_AWAY_ACTIVE mcast->fragmented 41
THROW_AWAY_ACTIVE mcast->fragmented 42
THROW_AWAY_ACTIVE mcast->fragmented 43
THROW_AWAY_ACTIVE mcast->fragmented 44
THROW_AWAY_ACTIVE mcast->fragmented 45
THROW_AWAY_ACTIVE mcast->fragmented 46
THROW_AWAY_ACTIVE mcast->fragmented 18
THROW_AWAY_ACTIVE mcast->fragmented 19
THROW_AWAY_ACTIVE mcast->fragmented 0
THROW_AWAY_ACTIVE mcast->fragmented 48
THROW_AWAY_ACTIVE mcast->fragmented 49
THROW_AWAY_ACTIVE mcast->fragmented 50
THROW_AWAY_ACTIVE mcast->fragmented 51
THROW_AWAY_ACTIVE mcast->fragmented 52
THROW_AWAY_ACTIVE mcast->fragmented 53
THROW_AWAY_ACTIVE mcast->fragmented 54
THROW_AWAY_ACTIVE mcast->fragmented 55
THROW_AWAY_ACTIVE mcast->fragmented 56
THROW_AWAY_ACTIVE mcast->fragmented 57
THROW_AWAY_ACTIVE mcast->fragmented 58
THROW_AWAY_ACTIVE mcast->fragmented 59
THROW_AWAY_ACTIVE mcast->fragmented 60
THROW_AWAY_ACTIVE mcast->fragmented 61
THROW_AWAY_ACTIVE mcast->fragmented 62
THROW_AWAY_ACTIVE mcast->fragmented 63
THROW_AWAY_ACTIVE mcast->fragmented 64
THROW_AWAY_ACTIVE mcast->fragmented 65
THROW_AWAY_ACTIVE mcast->fragmented 66
THROW_AWAY_ACTIVE mcast->fragmented 67
THROW_AWAY_ACTIVE mcast->fragmented 68
THROW_AWAY_ACTIVE mcast->fragmented 69
THROW_AWAY_ACTIVE mcast->fragmented 70
THROW_AWAY_ACTIVE mcast->fragmented 71
THROW_AWAY_ACTIVE mcast->fragmented 72
THROW_AWAY_ACTIVE mcast->fragmented 73
THROW_AWAY_ACTIVE mcast->fragmented 74
THROW_AWAY_ACTIVE mcast->fragmented 75
THROW_AWAY_ACTIVE mcast->fragmented 76
THROW_AWAY_ACTIVE mcast->fragmented 77
THROW_AWAY_ACTIVE mcast->fragmented 78
THROW_AWAY_ACTIVE mcast->fragmented 79
THROW_AWAY_ACTIVE mcast->fragmented 80
THROW_AWAY_ACTIVE mcast->fragmented 0
THROW_AWAY_ACTIVE mcast->fragmented 2
THROW_AWAY_ACTIVE mcast->fragmented 3
THROW_AWAY_ACTIVE mcast->fragmented 4
THROW_AWAY_ACTIVE mcast->fragmented 5
THROW_AWAY_ACTIVE mcast->fragmented 6
THROW_AWAY_ACTIVE mcast->fragmented 7
THROW_AWAY_ACTIVE mcast->fragmented 8
THROW_AWAY_ACTIVE mcast->fragmented 9
THROW_AWAY_ACTIVE mcast->fragmented 10
THROW_AWAY_ACTIVE mcast->fragmented 11
THROW_AWAY_ACTIVE mcast->fragmented 12
THROW_AWAY_ACTIVE mcast->fragmented 13
THROW_AWAY_ACTIVE mcast->fragmented 14
THROW_AWAY_ACTIVE mcast->fragmented 15
THROW_AWAY_ACTIVE mcast->fragmented 1
THROW_AWAY_ACTIVE mcast->fragmented 3
THROW_AWAY_ACTIVE mcast->fragmented 4
THROW_AWAY_ACTIVE mcast->fragmented 5
THROW_AWAY_ACTIVE mcast->fragmented 6
THROW_AWAY_ACTIVE mcast->fragmented 7
THROW_AWAY_ACTIVE mcast->fragmented 8
THROW_AWAY_ACTIVE mcast->fragmented 9
THROW_AWAY_ACTIVE mcast->fragmented 10
THROW_AWAY_ACTIVE mcast->fragmented 11
THROW_AWAY_ACTIVE mcast->fragmented 12
THROW_AWAY_ACTIVE mcast->fragmented 13
THROW_AWAY_ACTIVE mcast->fragmented 14
THROW_AWAY_ACTIVE mcast->fragmented 15
THROW_AWAY_ACTIVE mcast->fragmented 16
THROW_AWAY_ACTIVE mcast->fragmented 17
THROW_AWAY_ACTIVE mcast->fragmented 18
THROW_AWAY_ACTIVE mcast->fragmented 19
THROW_AWAY_ACTIVE mcast->fragmented 20
THROW_AWAY_ACTIVE mcast->fragmented 0
-----Original Message-----
From: Steven Dake [mailto:[email protected]]
Sent: Monday, April 13, 2009 2:56 PM
To: jay_chen
Cc: [email protected]
Subject: [Spam Mail] Re: [Openais] sync_in_process always return 1
On Mon, 2009-04-13 at 10:38 +0800, jay_chen wrote:
> Dear All:
>
> I am using 0.80.4 and I encounter a problem.
> I have two nodes on different devices and I checkpoint data from
> master to slave node.
> After they run for a long time, it always return SA_AIS_ERR_TRY_AGAIN
> on master. (which is doing checkpoint write operation periodically)
>
> Some observations:
> 1. sync_in_process( ) is always 1.
> 2. master & slave call sync_barrier_send( ) and return with res = 0
> (which means ok) 3. master call sync_deliver_fn( ) with slave nodeid
> only 4. slave call sync_deliver_fn with slave nodeid only 5. the state
> machine of sync remains in processing state forever
>
> Could anyody give me some hints to further investigate?
>
logs around the time this happens would be helpful.
Is the THROW_AWAY code in totempg throwing away the first message in the new
configuration?
what service is syncing when this barrier occurs?
what io load is taking place (transactions per second).
regards
-steve
> Thanks.
>
> Jay Chen.
> _______________________________________________
> Openais mailing list
> [email protected]
> https://lists.linux-foundation.org/mailman/listinfo/openais
_______________________________________________
Openais mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/openais