[gem5-users] Re: Slow checkpoint creation
This is weird; it never took me more than 20 seconds to generate a single checkpoint in gem5. I suspect most of the time is spent on draining the system. Would you be able to debug which draining method is stalling the checkpointing process? Kind Regards Giacomo From: Hanhwi Jang via gem5-users Date: Sunday, 13 November 2022 at 18:27 To: gem5-users@gem5.org Cc: Hanhwi Jang Subject: [gem5-users] Slow checkpoint creation Hi everyone, I recently returned to gem5 and tried to create SimPoints for SPEC and other workloads on ARM architecture. gem5 works great as usual, but I am curious why it takes 10~30 mins to write just a single checkpoint. The simulated system has 16 GB DRAM. Typically taking a snapshot of a similar system spends just a couple of minutes if we use VMware, QEMU, or other hypervisors. Is there anyone who dug into this issue? Thanks, Hanhwi IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ___ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org
[gem5-users] Re: Slow checkpoint creation
I currently use ARM-KVM with gem5, so reaching the point for checkpointing takes just a couple of minutes and is not a problem. Just writing a checkpoint takes 10 mins even on locally-mounted SSDs. Before digging into the issue, I just wanted to know if this is normal. Maybe I have to check my system configuration again. Thank you for your advice. Thanks, Hanhwi 2022년 11월 15일 (화) 오전 6:44, Eliot Moss 님이 작성: > On 11/14/2022 12:42 AM, Hanhwi Jang via gem5-users wrote: > > Thanks for your reply. > > > > Yes. I mean that; serializing all SimObjects and writing them on a disk. > > Does the checkpoint latency depend on the workload behavior? > > For GCC, the checkpoint takes around 10 mins for a single point,' but it > takes 3 mins for MCF. > > I am not sure this is normal. > > Not sure we're communicating. > > When starting from boot, it takes about 15mins to reach the point > where a checkpoint will be taken. *Saving* the checkpoint is quick. > Getting to the place *where* you checkpoint is what takes 15 mins > in this boot-up scenario. > > If it is really taking that long actually to save, I would say > something is wrong. Are you saving over a network, e.g., to some > remote mounted file system or something? That could affect things. > > Best - EM > ___ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org
[gem5-users] Re: Slow checkpoint creation
On 11/14/2022 12:42 AM, Hanhwi Jang via gem5-users wrote: Thanks for your reply. Yes. I mean that; serializing all SimObjects and writing them on a disk. Does the checkpoint latency depend on the workload behavior? For GCC, the checkpoint takes around 10 mins for a single point,' but it takes 3 mins for MCF. I am not sure this is normal. Not sure we're communicating. When starting from boot, it takes about 15mins to reach the point where a checkpoint will be taken. *Saving* the checkpoint is quick. Getting to the place *where* you checkpoint is what takes 15 mins in this boot-up scenario. If it is really taking that long actually to save, I would say something is wrong. Are you saving over a network, e.g., to some remote mounted file system or something? That could affect things. Best - EM ___ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org
[gem5-users] Re: Slow checkpoint creation
Thanks for your reply. Yes. I mean that; serializing all SimObjects and writing them on a disk. Does the checkpoint latency depend on the workload behavior? For GCC, the checkpoint takes around 10 mins for a single point,' but it takes 3 mins for MCF. I am not sure this is normal. Thanks, Hanhwi 2022년 11월 14일 (월) 오전 3:35, Eliot Moss 님이 작성: > On 11/13/2022 1:24 PM, Hanhwi Jang via gem5-users wrote: > > Hi everyone, > > > > I recently returned to gem5 and tried to create SimPoints for SPEC and > other workloads on ARM > > architecture. > > > > gem5 works great as usual, but I am curious why it takes 10~30 mins to > write just a single > > checkpoint. The simulated system has 16 GB DRAM. Typically taking a > snapshot of a similar system > > spends just a couple of minutes if we use VMware, QEMU, or other > hypervisors. > > > > Is there anyone who dug into this issue? > > If I am understanding what you're saying, the saving of the checkpoint > data is actually quick. it running through the whole OS boot procedure > that takes the 20 minutes +/- ... > > Eliot Moss > ___ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org
[gem5-users] Re: Slow checkpoint creation
On 11/13/2022 1:24 PM, Hanhwi Jang via gem5-users wrote: Hi everyone, I recently returned to gem5 and tried to create SimPoints for SPEC and other workloads on ARM architecture. gem5 works great as usual, but I am curious why it takes 10~30 mins to write just a single checkpoint. The simulated system has 16 GB DRAM. Typically taking a snapshot of a similar system spends just a couple of minutes if we use VMware, QEMU, or other hypervisors. Is there anyone who dug into this issue? If I am understanding what you're saying, the saving of the checkpoint data is actually quick. it running through the whole OS boot procedure that takes the 20 minutes +/- ... Eliot Moss ___ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org
