I am not quiet understand the operations of throw_away.
And I have a question. (maybe stupid)
Since different nodes uses different assemblys.
Why isn't a throw_away_mode for each node?
Jay...
-----Original Message-----
From: Steven Dake [mailto:[email protected]]
Sent: Tuesday, April 14, 2009 10:42 AM
To: jay_chen
Cc: [email protected]
Subject: Re: RE:Re: [Openais] sync_in_process always return 1
On Tue, 2009-04-14 at 09:18 +0800, jay_chen wrote:
> (send this mail again to remove "Spam Mail" on title)
>
> Dear steve:
> I add a printf in the following code.
> And I got many printf when the problem happen.
> I don't understand what is throw away.
> Is there any doc or explanation I could reference?
> Thanks.
>
The throw away logic was recently added to fix a defect, and apparently it
is buggy. Its purpose is to throw away one message that are fragmented
where some of the fragment contents have been lost. Right after a config
change, it should throw away all message fragments if there is a gap in the
message until it finds the start of a new message.
Do you have logs of the configuration changes that happen via clm.c?
I'd like to see the last two or three configuration changes.
> Jay...
>
> ====== code
> if (throw_away_mode == THROW_AWAY_ACTIVE) {
>
> printf("THROW_AWAY_ACTIVE mcast->fragmented %d\n",
> mcast->fragmented);
>
> /* Throw away the first msg block */
> if (mcast->fragmented == 0 || mcast->fragmented == 1) {
> throw_away_mode = THROW_AWAY_INACTIVE;
>
> assembly->index += msg_lens[0];
> iov_delv.iov_base =
> &assembly->data[assembly->index];
>
> ====== debug output
> sync_start_process
> sync_start_process call sync_barrier_send with res 0 THROW_AWAY_ACTIVE
> mcast->fragmented 150 THROW_AWAY_ACTIVE mcast->fragmented 151
> THROW_AWAY_ACTIVE mcast->fragmented 152 THROW_AWAY_ACTIVE
> mcast->fragmented 153 THROW_AWAY_ACTIVE mcast->fragmented 154
> THROW_AWAY_ACTIVE mcast->fragmented 155 THROW_AWAY_ACTIVE
> mcast->fragmented 156 THROW_AWAY_ACTIVE mcast->fragmented 157
> THROW_AWAY_ACTIVE mcast->fragmented 158 THROW_AWAY_ACTIVE
> mcast->fragmented 159 THROW_AWAY_ACTIVE mcast->fragmented 160
> THROW_AWAY_ACTIVE mcast->fragmented 161 THROW_AWAY_ACTIVE
> mcast->fragmented 162 THROW_AWAY_ACTIVE mcast->fragmented 163
> THROW_AWAY_ACTIVE mcast->fragmented 164 THROW_AWAY_ACTIVE
> mcast->fragmented 165 THROW_AWAY_ACTIVE mcast->fragmented 1
> THROW_AWAY_ACTIVE mcast->fragmented 3 THROW_AWAY_ACTIVE
> mcast->fragmented 4 THROW_AWAY_ACTIVE mcast->fragmented 5
> THROW_AWAY_ACTIVE mcast->fragmented 6 THROW_AWAY_ACTIVE
> mcast->fragmented 7 THROW_AWAY_ACTIVE mcast->fragmented 8
> THROW_AWAY_ACTIVE mcast->fragmented 9 THROW_AWAY_ACTIVE
> mcast->fragmented 10 THROW_AWAY_ACTIVE mcast->fragmented 11
> THROW_AWAY_ACTIVE mcast->fragmented 12 THROW_AWAY_ACTIVE
> mcast->fragmented 13 THROW_AWAY_ACTIVE mcast->fragmented 14
> THROW_AWAY_ACTIVE mcast->fragmented 15 THROW_AWAY_ACTIVE
> mcast->fragmented 16 THROW_AWAY_ACTIVE mcast->fragmented 17
> THROW_AWAY_ACTIVE mcast->fragmented 166 THROW_AWAY_ACTIVE
> mcast->fragmented 167 THROW_AWAY_ACTIVE mcast->fragmented 168
> THROW_AWAY_ACTIVE mcast->fragmented 169 THROW_AWAY_ACTIVE
> mcast->fragmented 170 THROW_AWAY_ACTIVE mcast->fragmented 171
> THROW_AWAY_ACTIVE mcast->fragmented 172 THROW_AWAY_ACTIVE
> mcast->fragmented 173 THROW_AWAY_ACTIVE mcast->fragmented 174
> THROW_AWAY_ACTIVE mcast->fragmented 175 THROW_AWAY_ACTIVE
> mcast->fragmented 176 THROW_AWAY_ACTIVE mcast->fragmented 177
> THROW_AWAY_ACTIVE mcast->fragmented 178 THROW_AWAY_ACTIVE
> mcast->fragmented 179 THROW_AWAY_ACTIVE mcast->fragmented 180
> THROW_AWAY_ACTIVE mcast->fragmented 181 THROW_AWAY_ACTIVE
> mcast->fragmented 18 THROW_AWAY_ACTIVE mcast->fragmented 19
> THROW_AWAY_ACTIVE mcast->fragmented 20 THROW_AWAY_ACTIVE
> mcast->fragmented 21 THROW_AWAY_ACTIVE mcast->fragmented 22
> THROW_AWAY_ACTIVE mcast->fragmented 23 THROW_AWAY_ACTIVE
> mcast->fragmented 24 THROW_AWAY_ACTIVE mcast->fragmented 25
> THROW_AWAY_ACTIVE mcast->fragmented 26 THROW_AWAY_ACTIVE
> mcast->fragmented 27 THROW_AWAY_ACTIVE mcast->fragmented 182
> THROW_AWAY_ACTIVE mcast->fragmented 183 THROW_AWAY_ACTIVE
> mcast->fragmented 184 THROW_AWAY_ACTIVE mcast->fragmented 185
> THROW_AWAY_ACTIVE mcast->fragmented 186 THROW_AWAY_ACTIVE
> mcast->fragmented 187 THROW_AWAY_ACTIVE mcast->fragmented 188
> THROW_AWAY_ACTIVE mcast->fragmented 189 THROW_AWAY_ACTIVE
> mcast->fragmented 190 THROW_AWAY_ACTIVE mcast->fragmented 191
> THROW_AWAY_ACTIVE mcast->fragmented 192 THROW_AWAY_ACTIVE
> mcast->fragmented 193 THROW_AWAY_ACTIVE mcast->fragmented 194
> THROW_AWAY_ACTIVE mcast->fragmented 195 THROW_AWAY_ACTIVE
> mcast->fragmented 196 THROW_AWAY_ACTIVE mcast->fragmented 197
> THROW_AWAY_ACTIVE mcast->fragmented 198 THROW_AWAY_ACTIVE
> mcast->fragmented 28 THROW_AWAY_ACTIVE mcast->fragmented 29
> THROW_AWAY_ACTIVE mcast->fragmented 30 THROW_AWAY_ACTIVE
> mcast->fragmented 31 THROW_AWAY_ACTIVE mcast->fragmented 32
> THROW_AWAY_ACTIVE mcast->fragmented 33 THROW_AWAY_ACTIVE
> mcast->fragmented 34 THROW_AWAY_ACTIVE mcast->fragmented 35
> THROW_AWAY_ACTIVE mcast->fragmented 36 THROW_AWAY_ACTIVE
> mcast->fragmented 199 THROW_AWAY_ACTIVE mcast->fragmented 37
> THROW_AWAY_ACTIVE mcast->fragmented 38 THROW_AWAY_ACTIVE
> mcast->fragmented 39 THROW_AWAY_ACTIVE mcast->fragmented 40
> THROW_AWAY_ACTIVE mcast->fragmented 200 THROW_AWAY_ACTIVE
> mcast->fragmented 201 THROW_AWAY_ACTIVE mcast->fragmented 202
> THROW_AWAY_ACTIVE mcast->fragmented 203 THROW_AWAY_ACTIVE
> mcast->fragmented 204 THROW_AWAY_ACTIVE mcast->fragmented 205
> THROW_AWAY_ACTIVE mcast->fragmented 206 THROW_AWAY_ACTIVE
> mcast->fragmented 207 THROW_AWAY_ACTIVE mcast->fragmented 208
> THROW_AWAY_ACTIVE mcast->fragmented 209 THROW_AWAY_ACTIVE
> mcast->fragmented 210 THROW_AWAY_ACTIVE mcast->fragmented 211
> THROW_AWAY_ACTIVE mcast->fragmented 212 THROW_AWAY_ACTIVE
> mcast->fragmented 213 THROW_AWAY_ACTIVE mcast->fragmented 214
> THROW_AWAY_ACTIVE mcast->fragmented 215 THROW_AWAY_ACTIVE
> mcast->fragmented 216 THROW_AWAY_ACTIVE mcast->fragmented 1
> THROW_AWAY_ACTIVE mcast->fragmented 3 THROW_AWAY_ACTIVE
> mcast->fragmented 4 THROW_AWAY_ACTIVE mcast->fragmented 5
> THROW_AWAY_ACTIVE mcast->fragmented 6 THROW_AWAY_ACTIVE
> mcast->fragmented 217 THROW_AWAY_ACTIVE mcast->fragmented 218
> THROW_AWAY_ACTIVE mcast->fragmented 219 THROW_AWAY_ACTIVE
> mcast->fragmented 220 THROW_AWAY_ACTIVE mcast->fragmented 221
> THROW_AWAY_ACTIVE mcast->fragmented 222 THROW_AWAY_ACTIVE
> mcast->fragmented 223 THROW_AWAY_ACTIVE mcast->fragmented 224
> THROW_AWAY_ACTIVE mcast->fragmented 225 THROW_AWAY_ACTIVE
> mcast->fragmented 226 THROW_AWAY_ACTIVE mcast->fragmented 227
> THROW_AWAY_ACTIVE mcast->fragmented 228 THROW_AWAY_ACTIVE
> mcast->fragmented 229 THROW_AWAY_ACTIVE mcast->fragmented 230
> THROW_AWAY_ACTIVE mcast->fragmented 231 THROW_AWAY_ACTIVE
> mcast->fragmented 232 THROW_AWAY_ACTIVE mcast->fragmented 233
> THROW_AWAY_ACTIVE mcast->fragmented 7 THROW_AWAY_ACTIVE
> mcast->fragmented 8 THROW_AWAY_ACTIVE mcast->fragmented 9
> THROW_AWAY_ACTIVE mcast->fragmented 234 THROW_AWAY_ACTIVE
> mcast->fragmented 235 THROW_AWAY_ACTIVE mcast->fragmented 236
> THROW_AWAY_ACTIVE mcast->fragmented 237 THROW_AWAY_ACTIVE
> mcast->fragmented 238 THROW_AWAY_ACTIVE mcast->fragmented 239
> THROW_AWAY_ACTIVE mcast->fragmented 240 THROW_AWAY_ACTIVE
> mcast->fragmented 241 THROW_AWAY_ACTIVE mcast->fragmented 242
> THROW_AWAY_ACTIVE mcast->fragmented 243 THROW_AWAY_ACTIVE
> mcast->fragmented 244 THROW_AWAY_ACTIVE mcast->fragmented 245
> THROW_AWAY_ACTIVE mcast->fragmented 246 THROW_AWAY_ACTIVE
> mcast->fragmented 247 THROW_AWAY_ACTIVE mcast->fragmented 248
> THROW_AWAY_ACTIVE mcast->fragmented 249 THROW_AWAY_ACTIVE
> mcast->fragmented 250 THROW_AWAY_ACTIVE mcast->fragmented 10
> THROW_AWAY_ACTIVE mcast->fragmented 11 THROW_AWAY_ACTIVE
> mcast->fragmented 12 THROW_AWAY_ACTIVE mcast->fragmented 13
> THROW_AWAY_ACTIVE mcast->fragmented 251 THROW_AWAY_ACTIVE
> mcast->fragmented 252 THROW_AWAY_ACTIVE mcast->fragmented 253
> THROW_AWAY_ACTIVE mcast->fragmented 254 THROW_AWAY_ACTIVE
> mcast->fragmented 255 THROW_AWAY_ACTIVE mcast->fragmented 1
> THROW_AWAY_ACTIVE mcast->fragmented 3 THROW_AWAY_ACTIVE
> mcast->fragmented 4 THROW_AWAY_ACTIVE mcast->fragmented 5
> THROW_AWAY_ACTIVE mcast->fragmented 6 THROW_AWAY_ACTIVE
> mcast->fragmented 7 THROW_AWAY_ACTIVE mcast->fragmented 8
> THROW_AWAY_ACTIVE mcast->fragmented 9 THROW_AWAY_ACTIVE
> mcast->fragmented 10 THROW_AWAY_ACTIVE mcast->fragmented 11
> THROW_AWAY_ACTIVE mcast->fragmented 12 THROW_AWAY_ACTIVE
> mcast->fragmented 14 THROW_AWAY_ACTIVE mcast->fragmented 13
> THROW_AWAY_ACTIVE mcast->fragmented 14 THROW_AWAY_ACTIVE
> mcast->fragmented 15 THROW_AWAY_ACTIVE mcast->fragmented 16
> THROW_AWAY_ACTIVE mcast->fragmented 17 THROW_AWAY_ACTIVE
> mcast->fragmented 18 THROW_AWAY_ACTIVE mcast->fragmented 19
> THROW_AWAY_ACTIVE mcast->fragmented 20 THROW_AWAY_ACTIVE
> mcast->fragmented 21 THROW_AWAY_ACTIVE mcast->fragmented 22
> THROW_AWAY_ACTIVE mcast->fragmented 23 THROW_AWAY_ACTIVE
> mcast->fragmented 24 THROW_AWAY_ACTIVE mcast->fragmented 25
> THROW_AWAY_ACTIVE mcast->fragmented 26 THROW_AWAY_ACTIVE
> mcast->fragmented 27 THROW_AWAY_ACTIVE mcast->fragmented 28
> THROW_AWAY_ACTIVE mcast->fragmented 29 THROW_AWAY_ACTIVE
> mcast->fragmented 15 THROW_AWAY_ACTIVE mcast->fragmented 16
> THROW_AWAY_ACTIVE mcast->fragmented 17 THROW_AWAY_ACTIVE
> mcast->fragmented 30 THROW_AWAY_ACTIVE mcast->fragmented 31
> THROW_AWAY_ACTIVE mcast->fragmented 32 THROW_AWAY_ACTIVE
> mcast->fragmented 33 THROW_AWAY_ACTIVE mcast->fragmented 34
> THROW_AWAY_ACTIVE mcast->fragmented 35 THROW_AWAY_ACTIVE
> mcast->fragmented 36 THROW_AWAY_ACTIVE mcast->fragmented 37
> THROW_AWAY_ACTIVE mcast->fragmented 38 THROW_AWAY_ACTIVE
> mcast->fragmented 39 THROW_AWAY_ACTIVE mcast->fragmented 40
> THROW_AWAY_ACTIVE mcast->fragmented 41 THROW_AWAY_ACTIVE
> mcast->fragmented 42 THROW_AWAY_ACTIVE mcast->fragmented 43
> THROW_AWAY_ACTIVE mcast->fragmented 44 THROW_AWAY_ACTIVE
> mcast->fragmented 45 THROW_AWAY_ACTIVE mcast->fragmented 46
> THROW_AWAY_ACTIVE mcast->fragmented 18 THROW_AWAY_ACTIVE
> mcast->fragmented 19 THROW_AWAY_ACTIVE mcast->fragmented 0
> THROW_AWAY_ACTIVE mcast->fragmented 48 THROW_AWAY_ACTIVE
> mcast->fragmented 49 THROW_AWAY_ACTIVE mcast->fragmented 50
> THROW_AWAY_ACTIVE mcast->fragmented 51 THROW_AWAY_ACTIVE
> mcast->fragmented 52 THROW_AWAY_ACTIVE mcast->fragmented 53
> THROW_AWAY_ACTIVE mcast->fragmented 54 THROW_AWAY_ACTIVE
> mcast->fragmented 55 THROW_AWAY_ACTIVE mcast->fragmented 56
> THROW_AWAY_ACTIVE mcast->fragmented 57 THROW_AWAY_ACTIVE
> mcast->fragmented 58 THROW_AWAY_ACTIVE mcast->fragmented 59
> THROW_AWAY_ACTIVE mcast->fragmented 60 THROW_AWAY_ACTIVE
> mcast->fragmented 61 THROW_AWAY_ACTIVE mcast->fragmented 62
> THROW_AWAY_ACTIVE mcast->fragmented 63 THROW_AWAY_ACTIVE
> mcast->fragmented 64 THROW_AWAY_ACTIVE mcast->fragmented 65
> THROW_AWAY_ACTIVE mcast->fragmented 66 THROW_AWAY_ACTIVE
> mcast->fragmented 67 THROW_AWAY_ACTIVE mcast->fragmented 68
> THROW_AWAY_ACTIVE mcast->fragmented 69 THROW_AWAY_ACTIVE
> mcast->fragmented 70 THROW_AWAY_ACTIVE mcast->fragmented 71
> THROW_AWAY_ACTIVE mcast->fragmented 72 THROW_AWAY_ACTIVE
> mcast->fragmented 73 THROW_AWAY_ACTIVE mcast->fragmented 74
> THROW_AWAY_ACTIVE mcast->fragmented 75 THROW_AWAY_ACTIVE
> mcast->fragmented 76 THROW_AWAY_ACTIVE mcast->fragmented 77
> THROW_AWAY_ACTIVE mcast->fragmented 78 THROW_AWAY_ACTIVE
> mcast->fragmented 79 THROW_AWAY_ACTIVE mcast->fragmented 80
> THROW_AWAY_ACTIVE mcast->fragmented 0 THROW_AWAY_ACTIVE
> mcast->fragmented 2 THROW_AWAY_ACTIVE mcast->fragmented 3
> THROW_AWAY_ACTIVE mcast->fragmented 4 THROW_AWAY_ACTIVE
> mcast->fragmented 5 THROW_AWAY_ACTIVE mcast->fragmented 6
> THROW_AWAY_ACTIVE mcast->fragmented 7 THROW_AWAY_ACTIVE
> mcast->fragmented 8 THROW_AWAY_ACTIVE mcast->fragmented 9
> THROW_AWAY_ACTIVE mcast->fragmented 10 THROW_AWAY_ACTIVE
> mcast->fragmented 11 THROW_AWAY_ACTIVE mcast->fragmented 12
> THROW_AWAY_ACTIVE mcast->fragmented 13 THROW_AWAY_ACTIVE
> mcast->fragmented 14 THROW_AWAY_ACTIVE mcast->fragmented 15
> THROW_AWAY_ACTIVE mcast->fragmented 1 THROW_AWAY_ACTIVE
> mcast->fragmented 3 THROW_AWAY_ACTIVE mcast->fragmented 4
> THROW_AWAY_ACTIVE mcast->fragmented 5 THROW_AWAY_ACTIVE
> mcast->fragmented 6 THROW_AWAY_ACTIVE mcast->fragmented 7
> THROW_AWAY_ACTIVE mcast->fragmented 8 THROW_AWAY_ACTIVE
> mcast->fragmented 9 THROW_AWAY_ACTIVE mcast->fragmented 10
> THROW_AWAY_ACTIVE mcast->fragmented 11 THROW_AWAY_ACTIVE
> mcast->fragmented 12 THROW_AWAY_ACTIVE mcast->fragmented 13
> THROW_AWAY_ACTIVE mcast->fragmented 14 THROW_AWAY_ACTIVE
> mcast->fragmented 15 THROW_AWAY_ACTIVE mcast->fragmented 16
> THROW_AWAY_ACTIVE mcast->fragmented 17 THROW_AWAY_ACTIVE
> mcast->fragmented 18 THROW_AWAY_ACTIVE mcast->fragmented 19
> THROW_AWAY_ACTIVE mcast->fragmented 20 THROW_AWAY_ACTIVE
> mcast->fragmented 0 -----Original Message-----
> From: Steven Dake [mailto:[email protected]]
> Sent: Monday, April 13, 2009 2:56 PM
> To: jay_chen
> Cc: [email protected]
> Subject: [Spam Mail] Re: [Openais] sync_in_process always return 1
>
> On Mon, 2009-04-13 at 10:38 +0800, jay_chen wrote:
> > Dear All:
> >
> > I am using 0.80.4 and I encounter a problem.
> > I have two nodes on different devices and I checkpoint data from
> > master to slave node.
> > After they run for a long time, it always return
> > SA_AIS_ERR_TRY_AGAIN on master. (which is doing checkpoint write
> > operation periodically)
> >
> > Some observations:
> > 1. sync_in_process( ) is always 1.
> > 2. master & slave call sync_barrier_send( ) and return with res = 0
> > (which means ok) 3. master call sync_deliver_fn( ) with slave nodeid
> > only 4. slave call sync_deliver_fn with slave nodeid only 5. the
> > state machine of sync remains in processing state forever
> >
> > Could anyody give me some hints to further investigate?
> >
> logs around the time this happens would be helpful.
>
> Is the THROW_AWAY code in totempg throwing away the first message in
> the new configuration?
>
> what service is syncing when this barrier occurs?
>
> what io load is taking place (transactions per second).
>
> regards
> -steve
>
> > Thanks.
> >
> > Jay Chen.
> > _______________________________________________
> > Openais mailing list
> > [email protected]
> > https://lists.linux-foundation.org/mailman/listinfo/openais
>
_______________________________________________
Openais mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/openais
