YANG Song (杨嵩)
IBM China System Technology Laboratory
Tel: 86-10-82452903
Email: yang...@cn.ibm.com
Address: Building 28, ZhongGuanCun Software Park,
No.8, Dong Bei Wang West Road, Haidian District Beijing 100193, PRC
北京市海淀区东北旺西路8号中关村软件园28号楼
邮编: 100193
----- Original message -----
From: Angelo Cavalcanti <angelo.cavalca...@gmail.com>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Cc:
Subject: Re: [xcat-user] [External] Netboot process stuck
Date: Wed, Mar 20, 2019 1:37 AM
Hi Song,The xCAT probe session output is below:# xcatprobe osdeploy -n gn001The install NIC in current server is p2p2 [INFO]All nodes to be deployed are valid [ OK ]-------------------------------------------------------------Start capturing every message during OS provision process....-------------------------------------------------------------[gn001] 10:04:13 Receive DHCPDISCOVER via p2p2[gn001] 10:04:13 Send DHCPOFFER on 10.1.2.1 back to 00:25:90:6c:a8:a2 via p2p2[gn001] 10:04:15 DHCPREQUEST for 10.1.2.1 (10.1.0.254) from 00:25:90:6c:a8:a2 via p2p2[gn001] 10:04:15 Send DHCPACK on 10.1.2.1 back to 00:25:90:6c:a8:a2 via p2p2[gn001] 10:04:15 Via TFTP download pxelinux.0[gn001] 10:04:15 Via TFTP download pxelinux.0[gn001] 10:04:15 Via TFTP download pxelinux.cfg/00000000-0000-0000-0000-0025906ca8a2[gn001] 10:04:15 Via TFTP download pxelinux.cfg/01-00-25-90-6c-a8-a2[gn001] 10:04:15 Via TFTP download pxelinux.cfg/0A010201[gn001] 10:04:15 Via TFTP download xcat/osimage/centos7.6-x86_64-netboot-gpu-compute/kernel[gn001] 10:04:16 Via TFTP download xcat/osimage/centos7.6-x86_64-netboot-gpu-compute/initrd-stateless.gz[gn001] 10:20:10 Receive DHCPDISCOVER via p2p2[gn001] 10:20:10 Send DHCPOFFER on 10.1.2.1 back to 00:25:90:6c:a8:a2 via p2p2[gn001] 10:20:10 DHCPREQUEST for 10.1.2.1 (10.1.0.254) from 00:25:90:6c:a8:a2 via p2p2[gn001] 10:20:10 Send DHCPACK on 10.1.2.1 back to 00:25:90:6c:a8:a2 via p2p2[gn001] 10:20:17 INFO =============deployment starting====================[gn001] 10:20:17 INFO =============deployment starting====================[gn001] 10:20:17 INFO Executing xcatroot to prepare for netbooting (dracut_33)...[gn001] 10:20:17 INFO Sending request to 10.1.0.254:3002 for changing status to netbooting...[gn001] 10:20:18 Node status is changed to netbooting[gn001] 10:20:18 INFO Downloading rootfs image from http://10.1.0.254:80//install/netboot/centos7.6/x86_64/gpu-...[gn001] 10:20:17 INFO Executing xcatroot to prepare for netbooting (dracut_33)...[gn001] 10:20:17 INFO Sending request to 10.1.0.254:3002 for changing status to netbooting...[gn001] 10:20:18 INFO Downloading rootfs image from http://10.1.0.254:80//install/netboot/centos7.6/x86_64/gpu-...[gn001] 10:20:18 Via HTTP get //install/netboot/centos7.6/x86_64/gpu-compute/rootimg.cpio.gz[gn001] 10:20:27 INFO Setting up RAM-root tmpfs on downloaded rootimg.cpio.[gz/xz]...[gn001] 10:20:27 INFO Setting up RAM-root tmpfs on downloaded rootimg.cpio.[gz/xz]...[gn001] 10:20:47 INFO Exiting xcatroot...[gn001] 10:20:47 INFO Exiting xcatroot...And below, the piece of the DHCP lease file for the node:host gn001 {
dynamic;
hardware ethernet 00:25:90:6c:a8:a2;
uid 00:25:90:6c:a8:a2;
fixed-address 10.1.2.1;
supersede server.ddns-hostname = "gn001";
supersede host-name = "gn001";
if option vendor-class-identifier = "ScaleMP" {
supersede server.filename = "vsmp/pxelinux.0";
} else {
supersede server.filename = "pxelinux.0";
}
}
Regards--Em sáb, 16 de mar de 2019 às 09:38, Angelo Cavalcanti <angelo.cavalca...@gmail.com> escreveu:1. The status is "powering-on"2. Yes, the issue happens in the same node3. Ok. I will send the xCAT-probe output sessionEm sex, 15 de mar de 2019 às 07:37, Song BJ Yang <yang...@cn.ibm.com> escreveu:Hi,If the console output covers the whole process, seems the the initrd boot up process did not reach the rootimg download phase. And there is a `[2019-03-14T10:20:39-03:00] [ 37.280041] systemd-fstab-generator[261]: Could not find a root= entry on the kernel command line.`,several questions:1. what is the node status: `lsdef <node> -i status,statustime`? is it changed to "netbooting"?2. did you provision a batch of nodes with the same osimage? did the issue always appear on the same node?3. please install xCAT-probe on you MN, run `xcatprobe xcatmn` to check if any configuration issue.and watch `xcatprobe osdeploy -n <failing node>` in 1 terminal session, and then kick off provision. You will find the provision progress in the xcatprobe session. Please provide that session output.
------------------------------------------------------------------------------YANG Song (杨嵩)
IBM China System Technology Laboratory
Tel: 86-10-82452903
Email: yang...@cn.ibm.com
Address: Building 28, ZhongGuanCun Software Park,
No.8, Dong Bei Wang West Road, Haidian District Beijing 100193, PRC
北京市海淀区东北旺西路8号中关村软件园28号楼
邮编: 100193----- Original message -----
From: Angelo Cavalcanti <angelo.cavalca...@gmail.com>
To: Song BJ Yang <yang...@cn.ibm.com>
Cc: xcat-user@lists.sourceforge.net
Subject: Re: [xcat-user] [External] Netboot process stuck
Date: Fri, Mar 15, 2019 10:37 AM
Thanks Song,I added the following kernel parameters:debug ignore_loglevel log_buf_len=10M print_fatal_signals=1The console output file is attached. I noticed that the machine's devices were not found in the udev database.Regards,Em qui, 14 de mar de 2019 06:41, Song BJ Yang <yang...@cn.ibm.com> escreveu:Hi,We encountered a similar issue https://github.com/xcat2/xcat-core/issues/274 , but in this case the console uncovered the root cause.However, your console output does not show why the boot up process hang. I suggest you add more verbose output during boot up, this is a reference dochttps://www.askapache.com/linux/linux-debugging/ on how to get more debug info during kernel boot up.To apply the kernel options during diskless kernel boot up, you can leverage the `addkcmdline` attribute, an example for addkcmdline usage ,chdef mid05tor12cn05 addkcmdline="debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug"good luck------------------------------------------------------------------------------
YANG Song (杨嵩)
IBM China System Technology Laboratory
Tel: 86-10-82452903
Email: yang...@cn.ibm.com
Address: Building 28, ZhongGuanCun Software Park,
No.8, Dong Bei Wang West Road, Haidian District Beijing 100193, PRC
北京市海淀区东北旺西路8号中关村软件园28号楼
邮编: 100193----- Original message -----
From: Jarrod Johnson <jjohns...@lenovo.com>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Cc:
Subject: Re: [xcat-user] [External] Netboot process stuck
Date: Thu, Mar 14, 2019 5:33 AM
What does the boot kernel get command line wise (e.g. /tftpboot/xcat/xnba/nodes/<nodename>)
From: Angelo Cavalcanti <angelo.cavalca...@gmail.com>
Sent: Wednesday, March 13, 2019 4:23 PM
To: xcat-user@lists.sourceforge.net
Subject: [External] [xcat-user] Netboot process stuck
Hi everyone,
I've setup various nodes to netboot image but one of them stuck in boot process, below:
[ 485.127448] systemd[1]: Reached target Sockets.
[ 515.134248] systemd[1]: Started Journal Service.
[ 635.252091] RPC: Registered named UNIX socket transport module.
[ 635.258163] RPC: Registered udp transport module.
[ 635.263004] RPC: Registered tcp transport module.
[ 635.267847] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 845.308189] pps_core: LinuxPPS API ver. 1 registered
[ 845.313333] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giome...@linux.it>
[ 845.325629] PTP clock support registered
[ 845.332035] dca service started, version 1.12.1
[ 845.343852] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 845.349585] mlx4_core: Initializing 0000:04:00.0
[ 845.364042] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
[ 845.368351] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[ 845.368355] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst
[ 845.380249] scsi host0: ahci
[ 845.380938] scsi host1: ahci
[ 845.382757] scsi host2: ahci
[ 845.383161] scsi host3: ahci
[ 845.386805] scsi host4: ahci
[ 845.387739] scsi host5: ahci
[ 845.387852] ata1: SATA max UDMA/133 abar m2048@0xde100000 port 0xde100100 irq 39
[ 845.387855] ata2: SATA max UDMA/133 abar m2048@0xde100000 port 0xde100180 irq 39
[ 845.387857] ata3: SATA max UDMA/133 abar m2048@0xde100000 port 0xde100200 irq 39
[ 845.387860] ata4: SATA max UDMA/133 abar m2048@0xde100000 port 0xde100280 irq 39
[ 845.387862] ata5: SATA max UDMA/133 abar m2048@0xde100000 port 0xde100300 irq 39
[ 845.387865] ata6: SATA max UDMA/133 abar m2048@0xde100000 port 0xde100380 irq 39
[ 845.451810] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 845.511186] igb 0000:81:00.0: added PHC on eth0
[ 845.515869] igb 0000:81:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 845.522896] igb 0000:81:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:25:90:6c:a8:a2
[ 845.530236] igb 0000:81:00.0: eth0: PBA No: 104900-000
[ 845.535510] igb 0000:81:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 845.598681] igb 0000:81:00.1: added PHC on eth1
[ 845.603379] igb 0000:81:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 845.610404] igb 0000:81:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:25:90:6c:a8:a3
[ 845.617768] igb 0000:81:00.1: eth1: PBA No: 104900-000
[ 845.623050] igb 0000:81:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 845.694180] ata4: SATA link down (SStatus 0 SControl 300)
[ 845.699730] ata3: SATA link down (SStatus 0 SControl 300)
[ 845.705276] ata2: SATA link down (SStatus 0 SControl 300)
[ 845.710813] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 845.717162] ata5: SATA link down (SStatus 0 SControl 300)
[ 845.722738] ata6: SATA link down (SStatus 0 SControl 300)
[ 845.728523] ata1.00: ATA-8: ST91000640NS, SN03, max UDMA/133
[ 845.734330] ata1.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 31/32)
[ 845.742162] ata1.00: configured for UDMA/133
[ 845.746833] scsi 0:0:0:0: Direct-Access ATA ST91000640NS SN03 PQ: 0 ANSI: 5
[ 845.791663] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 845.799600] sd 0:0:0:0: [sda] Write Protect is off
[ 845.804545] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 845.820557] sda: sda1 sda2
[ 845.823892] sd 0:0:0:0: [sda] Attached SCSI disk
[ 851.888095] mlx4_core 0000:04:00.0: Old device ETS support detected
[ 851.894490] mlx4_core 0000:04:00.0: Consider upgrading device FW.
[ 852.632012] mlx4_core 0000:04:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s
[ 852.640265] mlx4_core 0000:04:00.0: PCIe link width is x8, device supports x8
[ 852.787509] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 945.585085] igb 0000:81:00.0: changing MTU from 1500 to 2044
[ 949.821377] igb 0000:81:00.0 enp129s0f0: igb: enp129s0f0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 950.903525] random: crng init done
Notice that the boot process is slow.
The machine has the following configuration:
2x Intel Xeon E5-2670
256GB RAM
Motherboard Supermicro X9DRG-HF
HDD 1TB
Mellanox Infiniband ConnectX-3 card (MT27500)
GPGPU nVidia Tesla M2075
I removed all off-board cards and HDD. The boot process stays stuck in the same stage. I installed CentOS 7 minimal ISO on HDD and the problem did not occur.
Regards,
--
Angelo Cavalcanti
br.linkedin.com/in/angelocr_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
