Marcelo Tosatti wrote:
> Hi David,
>
> On Fri, Jul 11, 2008 at 03:18:54PM -0600, David S. Ahern wrote:
>> What's the status with this for full virt guests?
>
> The consensus seems to be that fullvirt guests need assistance from the
> management app (libvirt) to have boosted priority during their boot
> stage, so loops_per_jiffy calibration can be performed safely. As Daniel
> pointed out this is tricky because you can't know for sure how long the
> boot up will take, if for example PXE is used.
I boosted the priority of the guest to investigate an idea that maybe
some startup calibration in the guest was off slightly leading to
systematic drifting. I was on vacation last week and I am still catching
up with traffic on this list. I just happened to see your first message
with the panic which aligned with one of my tests.
>
> Glauber is working on some paravirt patches to remedy the situation.
>
> But loops_per_jiffy is not directly related to clock drifts, so this
> is a separate problem.
>
>> I am still seeing systematic time drifts in RHEL 3 and RHEL4 guests
>> which I've been digging into it the past few days.
>
> All time drift issues we were aware of are fixed in kvm-70. Can you
> please provide more details on how you see the time drifting with
> RHEL3/4 guests? It slowly but continually drifts or there are large
> drifts at once? Are they using TSC or ACPIPM as clocksource?
The attached file shows one example of the drift I am seeing. It's for a
4-way RHEL3 guest started with 'nice -20'. After startup each vcpu was
pinned to a physical cpu using taskset. The only activity on the host is
this one single guest; the guest is relatively idle -- about 4% activity
(~1% user, ~3% system time). Host is synchronized to an ntp server; the
guest is not. The guest is started with the -localtime parameter. From
the file you can see the guest gains about 1-2 seconds every 5 minutes.
Since it's a RHEL3 guest I believe the PIT is the only choice (how to
confirm?), though it does read the TSC (ie., use_tsc is 1).
>
> Also, most issues we've seen could only be replicated with dyntick
> guests.
>
> I'll try to reproduce it locally.
>
>> In the course of it I have been launching guests with boosted priority
>> (both nice -20 and realtime priority (RR 1)) on a nearly 100% idle
>> host.
>
> Can you also see wacked bogomips without boosting the guest priority?
The wacked bogomips only shows up when started with real-time priority.
With the 'nice -20' it's sane and close to what the host shows.
As another data point I restarted the RHEL3 guest using the -no-kvm-pit
and -tdf options (nice -20 priority boost). After 22 hours of uptime,
the guest is 29 seconds *behind* the host. Using the in-kernel pit the
guest time is always fast compared to the host.
I've seen similar drifting in RHEL4 guests, but I have not spent as much
time investigating it yet. On ESX adding clock=pit to the boot
parameters for RHEL4 guests helps immensely.
david
host-dt host time guest time guest-host-diff
300 1215748151 1215748262 111
301 1215748452 1215748563 111
300 1215748752 1215748865 113
300 1215749052 1215749165 113
300 1215749352 1215749466 114
301 1215749653 1215749768 115
300 1215749953 1215750069 116
300 1215750253 1215750369 116
300 1215750553 1215750671 118
301 1215750854 1215750972 118
300 1215751154 1215751273 119
300 1215751454 1215751575 121
300 1215751754 1215751875 121
301 1215752055 1215752176 121
300 1215752355 1215752477 122
300 1215752655 1215752780 125
300 1215752955 1215753083 128
301 1215753256 1215753385 129
300 1215753556 1215753686 130
300 1215753856 1215753988 132
300 1215754156 1215754289 133
301 1215754457 1215754592 135
300 1215754757 1215754894 137
300 1215755057 1215755198 141
300 1215755357 1215755499 142
301 1215755658 1215755799 141
300 1215755958 1215756101 143
300 1215756258 1215756402 144
300 1215756558 1215756702 144
301 1215756859 1215757005 146
300 1215757159 1215757307 148
300 1215757459 1215757609 150
301 1215757760 1215757910 150
300 1215758060 1215758211 151
300 1215758360 1215758515 155
300 1215758660 1215758816 156
301 1215758961 1215759118 157
300 1215759261 1215759418 157
300 1215759561 1215759720 159
300 1215759861 1215760022 161
301 1215760162 1215760323 161
300 1215760462 1215760625 163
300 1215760762 1215760927 165
300 1215761062 1215761229 167
301 1215761363 1215761532 169
300 1215761663 1215761833 170
300 1215761963 1215762136 173
300 1215762263 1215762439 176
301 1215762564 1215762741 177
300 1215762864 1215763043 179
300 1215763164 1215763348 184
300 1215763464 1215763649 185
301 1215763765 1215763950 185
300 1215764065 1215764251 186
300 1215764365 1215764552 187
300 1215764665 1215764854 189
301 1215764966 1215765154 188
300 1215765266 1215765455 189
300 1215765566 1215765757 191
300 1215765866 1215766058 192
300 1215766166 1215766358 192
301 1215766467 1215766666 199
300 1215766767 1215766966 199
300 1215767067 1215767269 202
300 1215767367 1215767571 204
301 1215767668 1215767871 203
300 1215767968 1215768173 205
300 1215768268 1215768476 208
300 1215768568 1215768777 209
301 1215768869 1215769078 209
300 1215769169 1215769379 210
300 1215769469 1215769682 213
300 1215769769 1215769984 215
301 1215770070 1215770286 216
300 1215770370 1215770589 219
300 1215770670 1215770890 220
301 1215770971 1215771191 220
300 1215771271 1215771496 225
300 1215771571 1215771798 227
300 1215771871 1215772100 229
301 1215772172 1215772403 231
300 1215772472 1215772704 232
300 1215772772 1215773007 235
300 1215773072 1215773307 235
301 1215773373 1215773613 240
300 1215773673 1215773916 243
300 1215773973 1215774220 247
300 1215774273 1215774520 247
301 1215774574 1215774822 248
300 1215774874 1215775122 248
300 1215775174 1215775425 251
300 1215775474 1215775725 251
301 1215775775 1215776027 252
300 1215776075 1215776327 252
300 1215776375 1215776628 253
300 1215776675 1215776928 253
301 1215776976 1215777230 254
300 1215777276 1215777531 255
300 1215777576 1215777833 257
300 1215777876 1215778136 260
301 1215778177 1215778435 258
300 1215778477 1215778736 259
300 1215778777 1215779037 260
300 1215779077 1215779337 260
301 1215779378 1215779640 262
300 1215779678 1215779941 263
300 1215779978 1215780241 263
300 1215780278 1215780544 266
301 1215780579 1215780843 264
300 1215780879 1215781146 267
300 1215781179 1215781448 269
300 1215781479 1215781749 270
301 1215782269 1215782544 275
300 1215782569 1215782844 275
300 1215782869 1215783145 276
300 1215783169 1215783447 278
301 1215783470 1215783749 279
300 1215783770 1215784050 280
300 1215784070 1215784352 282
300 1215784370 1215784654 284
301 1215784671 1215784955 284
300 1215784971 1215785258 287
300 1215785271 1215785559 288
300 1215785571 1215785861 290
301 1215785872 1215786162 290
300 1215786172 1215786463 291
300 1215786472 1215786767 295
300 1215786772 1215787068 296
301 1215787073 1215787369 296
300 1215787373 1215787670 297
