Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue

Thomas HUMMEL Thu, 03 Jun 2021 04:13:00 -0700

Thanks a lot Jarrod.

Well I guess indeed, for my use case the old xNBA seesm the way to go.

I should have done it (and still can) more cleanly using the RPM.Problem is though that any xCAT upgrade would pull the new rpm.


--
Thomas HUMMEL


On 03/06/2021 12:57, Jarrod Johnson wrote:

The slowness may relate to the fact it would be initializing *every* nic in the 
system, regardless of the one that booted, it could also relate to features, 
e.g. HTTPS can in some case cause init to be painfully slow depending on what 
entropy source it latches onto, and will do this initialization even if never 
used. I haven't actually booted the xnba to see the behavior personally.

My info about snponly.efi and ipxe.efi is intended mostly the benefit of 
whoever is building the xnba rpms. I'm a bit rusty and don't quite see how the 
build is currently managed though on that front.  I suspect the only difference 
between iPXE and xNBA is the user class identifier.  I've been mostly building 
for confluent, where we just use plain iPXE and if we do care about firmware 
ipxe missing features at some point, will use the feature block of iPXE 
requests to detect HTTP presence or absence instead of relying upon the user 
class identifier.

Xnba.kpxe is for 'legacy' bios boot rather than uefi, ipxe would call it 
'undionly.kpxe', and is not offered to UEFI boots.

One theory I have on what could go wrong is that when you chainboot, then the 
UEFI network stack is concurrently active with whatever iPXE stack is doing. 
The iPXE codebase may be changing things at the same time the firmware's code 
is also changing things . So an update to the UEFI generic network stack may 
produce better compatibility with whatever the ipxe codebase is doing, even if 
ipxe is to some extent trying to directly use the device.   Snponly.efi may be 
easier to live with, since all access to the hardware is arbitrated by a single 
set of code and there would be reduced potential for surprises.  Perhaps steps 
are taken that *should* suppress the uefi network stack, but somehow don't work 
consistently (maybe because the older firmware, didn't implement them 
correctly, or an unexpected gap in operation, hard to speculate). Or it could 
simply be that non-boot portion of the intel firmware that the new driver 
interacts with had bugs.

Again, I have only really seen this list activity to go on, so this is 
speculation without actually seeing the situation first hand. I think a rebuild 
of xnba-undi rpm with snponly.efi is in order, and if HTTPS is enabled, 
probably remove it for transfer size and initialization time benefits (the 
binary is tftp transferred, so security is not in the cards as a would-be http 
attack could simply freely modify the tftp data and change the cert validation 
freely).  If I were targeting security, UEFI enrolled certificates with UEFI 
native HTTPS boot would be a better cross-vendor approach.

-----Original Message-----
From: Thomas HUMMEL <thomas.hum...@pasteur.fr>
Sent: Thursday, June 3, 2021 5:24 AM
To: xcat-user@lists.sourceforge.net
Subject: Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue

Hello,

got it. This could explain (ipxe.efi case) the long initializing time as well, 
right ?

But in 
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=http*3A*2F*2Fxcat.org*2Ffiles*2Fxcat*2Frepos*2Fyum*2Fxcat-dep*2Frh8*2Fx86_64*2F&amp;data=04*7C01*7Cjjohnson2*40lenovo.com*7C5111016cfe4c415c61bc08d926718d4f*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C637583091425208346*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&amp;sdata=ofxojrtbeVSzj6OxsDgkPeoqf2uZvngki1dBe8b*2B2Ok*3D&amp;reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwTBpFCwcg$
  I only see

xnba-undi-1.0.3-131028.noarch.rpm and xnba-undi-1.20.1-0.noarch.rpm

So I guess you imply that if one wants eht snp version it has to built it 
himself, correct ?

Also, what is xnba.kpxe ?

What's strange though is that, for me it did or did not work across exact same 
nics (Intel X550 chipset) until I flashed the PXE and EFI Option ROM boot on 
the ones not working. That would rather happen with the snponly.efi case, 
wouldn't it ?

Thanks for your help

--
Thomas HUMMEL


On 02/06/2021 20:14, Jarrod Johnson wrote:

The iPXE codebase has code for things like UNDI ('legacy' bios generic
network stack) and SNP/NII for UEFI.  As far as code running targeting
these, the nic vendor shouldn't matter, because they abstract away
their differences in their uefi driver.

The ipxe code also strives to port in specific drivers, that bypass
the generic and target the network hardware more like linux would.
iPXE does this so it could boot from usb key or actually get flashed
and *be*  the firmware driver of an adapter card.

When building ipxe from source, you specify your target name:
make bin-x864_64-efi/ipxe.efi
That builds an ipxe with every driver bundled in and tries to use
specific driver for each possible device.

make bin-x86_64-efi/snponly.efi
Builds a binary suitable only for chainloader, delegating the network
hardware operation to the implementation provided by the firmware
rather than trying to do direct driver binding.

With default config options:
jjohnson2@wirbelwind:~/devel/ipxe/src$ ls -lh bin-x86_64-efi/*.efi
-rw-rw-r-- 1 jjohnson2 jjohnson2 1.1M Jun  2 14:13
bin-x86_64-efi/ipxe.efi
-rw-rw-r-- 1 jjohnson2 jjohnson2 234K Jun  2 14:13
bin-x86_64-efi/snponly.efi The snponly.efi is much smaller (my
production build is smaller still, I think my config drops some
features that I don't use)

----------------------------------------------------------------------
--
*From:* Thomas HUMMEL <thomas.hum...@pasteur.fr>
*Sent:* Wednesday, June 2, 2021 1:14 PM
*To:* xcat-user@lists.sourceforge.net
<xcat-user@lists.sourceforge.net>
*Subject:* Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue
Hello,

thanks for your answer.

Let me get this straight: basically you're saying that xnba can either
use ipxe.efi which would somehow user its own nic driver vs
snponly.efi which would rely on the built-in nic driver or something like this ?

I did indeed substitute xnba.efi with the one coming from the
xnba-undi-1.0.3 package (just the file, which is a dirty fix I know).

Thanks for your help

--
TH

On 02/06/2021 18:36, Jarrod Johnson wrote:

I decided to take a peek and note that the efi binary is pretty
large, so I guess it built ipxe.efi instead of 'snponly.efi', which
may complicate uefi boots (it at least means having to tftp down 10x
as much content before http can begin, which somewhat reduces the
point of ipxe chain booting).  ipxe.efi will try to use the 'good'
drivers for every network device while snponly.efi will try to hook
the generic uefi protocols provided by the devices.

When the 'good' drivers work, it generally does have somewhat better
performance, but I have generally better luck with the snponly.efi builds.

Might want to roll back to
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fxca__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRmMKionQ$
t.org%2Ffiles%2Fxcat%2Frepos%2Fyum%2F2.16%2Fxcat-dep%2Fxnba-undi-1.0.
3-131028.noarch.rpm&amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C51110
16cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C
0%7C637583091425208346%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=QY%2BBb
K3K6DyVA8%2B2xZxj9frwSgfT7ZupT%2BInIozW2%2FU%3D&amp;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Fxcat.org*2Ffiles*2Fxcat*2Frepos*2Fyum*2F2
.16*2Fxcat-dep*2Fxnba-undi-1.0.3-131028.noarch.rpm%26amp%3Bdata%3D04*7
C01*7Cjjohnson2*40lenovo.com*7C670c545b49d9431173a608d925ea22b8*7C5c7d
0b28bdf8410caa934df372b16203*7C1*7C0*7C637582509812053894*7CUnknown*7C
TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
I6Mn0*3D*7C1000%26amp%3Bsdata%3DNMbStO8pBhneWbPy091jtS5LW7qtrax4kaDinc
BYtuw*3D%26amp%3Breserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!JFd
NOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd1KGIY3Hs
iUf_MIOvG6ZCQ%24&amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C5111016cf
e4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C6
37583091425208346%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=hapVVnmQNalrUq
7iNu%2FKTwZquLTE6jn1TCDbA7UCTv0%3D&amp;reserved=0>

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fur__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ$
ldefense.com%2Fv3%2F__https%3A%2F%2Fxcat.org%2Ffiles%2Fxcat%2Frepos%2
Fyum%2F2.16%2Fxcat-dep%2Fxnba-undi-1.0.3-131028.noarch.rpm__%3B!!JFdN
OqOXpB6UZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o
7jLMfjioNLUlQ%24&amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C5111016c
fe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7
C637583091425208346%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI
joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=0YzQGXG1rw
dHeAqmiSUKqc%2BK600PqZwxkVl8yv5pZVY%3D&amp;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fxca
t.org*2Ffiles*2Fxcat*2Frepos*2Fyum*2F2.16*2Fxcat-dep*2Fxnba-undi-1.0.3
-131028.noarch.rpm__*3B!!JFdNOqOXpB6UZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce
4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfjioNLUlQ*24%26amp%3Bdata%3D04*7C01*7C
jjohnson2*40lenovo.com*7C670c545b49d9431173a608d925ea22b8*7C5c7d0b28bd
f8410caa934df372b16203*7C1*7C0*7C637582509812053894*7CUnknown*7CTWFpbG
Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*
3D*7C1000%26amp%3Bsdata%3DDASb7ZFfbKRmlH*2BcrXAet806ZvuLhuVrGuPxiw9KSg
A*3D%26amp%3Breserved%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl
JQ!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd1
KGIY3HsiUf_MIlFhVk5g%24&amp;data=04%7C01%7Cjjohnson2%40lenovo.com%7C51
11016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b16203%7C1%
7C0%7C637583091425218340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=S39FzyD
JRlsMPFe7MM15zHMjSS7vOLUrZ%2BidW5HooJo%3D&amp;reserved=0>> for

now, might be also interesting to build an snponly.efi instead of
ipxe.efi for a test..

---------------------------------------------------------------------
---
*From:* Thomas HUMMEL <thomas.hum...@pasteur.fr>
*Sent:* Wednesday, June 2, 2021 10:52 AM
*To:* xcat-user@lists.sourceforge.net
<xcat-user@lists.sourceforge.net>
*Subject:* [External] Re: [xcat-user] xCAT 2.16.2 new xNBA issue So,
basically, for me:

Supermicro H12DST-B with Intel 10G X550T (rev 01) and BIOS configured
in full UEFI mode with new xNBA:

a) does not boot if FW 1.93 and PXE version 2.4.16

b) does boot if PXE upgraded to 2.4.44 but after what seems an
anormaly long time spent in that stage:


        downloading NBP file : 10s
        xNBA initializing devices : 15s
        start of osimage fetch : 15s later

whereas regardless of PXE version, with old xNBA systems boot and
spend max 1s in xNBA stage.

What do you think about it ?

Thanks for your help

--
Thomas HUMMEL

On 02/06/2021 14:34, Thomas HUMMEL wrote:

Hello, thanks for your answer.

On 01/06/2021 22:42, Nathan A Besaw wrote:

xNBA 1.20.1 is almost identical to iPXE 1.20.1 with minimal
customization.


I'm by no means a PXE expert : am I right assuming iPXE and/or xNBA
use some PXE firmware built in into some EEPROM on the nic or board ?

Are there any differences in the BIOS boot mode settings between
the servers that are working and the servers that are failing?
Legacy mode vs Legacy + UEFI mode vs UEFI only mode?


No. They both boot in full UEFI mode (no legacy, no dual). Some may
have a different version but the issue occurs across the same BIOS version too.

Do you have better results if you downgrade xNBA to the previous version?


I indeed manage to workaround the issue by savagely replacing
xnba.efi by the one I extracted from the 2.15 branch, which I
believe was already xnba-undi-1.0.3-131028.noarch.rpm

Supermicro provided me as well with:

- nic EEPROM new firmare (I did not use nor flash)
- new PXE firmware which I flashed: this made the new xNBA works as
expected.

They stated that:

Batch 1 was shipped with a different EEPROM and PXE option ROIM
version then Batch 2 and 3.
Batch one has FW 1.93 and PXE version 2.4.16 while batch 2 and 3
will have FW 2.2 and PXE version 2.4.44

I still have to test flashing this new PXE fw to a batch2 node
(which worked with the new xNBA but spent >30s in it) and see if it
fixes the slow initialization.

Thanks

--
Thomas HUMMEL


_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fur__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ$
ldefense.com%2Fv3%2F__https%3A%2F%2Flists.sourceforge.net%2Flists%2F
listinfo%2Fxcat-user__%3B!!JFdNOqOXpB6UZW0!7CP7jiJu_SapsF7A5b-99TpS0
lU01tKQJu5BHi7YpIvbH3ekgOyxSZXEftL6i0z5gH4nOA%24&amp;data=04%7C01%7C
jjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28
bdf8410caa934df372b16203%7C1%7C0%7C637583091425218340%7CUnknown%7CTW
FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC
I6Mn0%3D%7C1000&amp;sdata=Hx%2FepCgCuSbKBQfiJi1BlEhgY9KQJZjR7drqFNAs
zNc%3D&amp;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Flis
ts.sourceforge.net*2Flists*2Flistinfo*2Fxcat-user__*3B!!JFdNOqOXpB6UZW
0!7CP7jiJu_SapsF7A5b-99TpS0lU01tKQJu5BHi7YpIvbH3ekgOyxSZXEftL6i0z5gH4n
OA*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b49d943
1173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C637582
509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DMIQ*2BZeb31pZ
1X*2Ba*2FarIbGWyh8*2FoxUMyj3f7H9RLu7rE*3D%26amp%3Breserved%3D0__%3BJSU
lJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_
sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MIVV0TrcQ%24&amp;data=04
%7C01%7Cjjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c
7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583091425218340%7CUnknown%
7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
VCI6Mn0%3D%7C1000&amp;sdata=LiTRzrueLyI%2FYZU3qWTIcRM3opt5bWj2oTAgYjUq
2hA%3D&amp;reserved=0>

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fur__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ$
ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo
k.com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2F
lists.sourceforge.net*2Flists*2Flistinfo*2Fxcat-user__*3B!!JFdNOqOXpB
6UZW0!7CP7jiJu_SapsF7A5b-99TpS0lU01tKQJu5BHi7YpIvbH3ekgOyxSZXEftL6i0z
5gH4nOA*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7Cf83e67c8
7894408a53e708d925d63f3e*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7
C637582424394899812*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI
joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DMv26
iG1vAmV6VkL9T2y9y7mCw2HynRccrgLEmkI7qPE*3D%26amp%3Breserved%3D0__%3BJ
SUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6UZW0!6qpRG5NAI9z1J0vKA
jYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfhpT0Lceg%24&amp;data=04%
7C01%7Cjjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c
7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583091425218340%7CUnknown
%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
JXVCI6Mn0%3D%7C1000&amp;sdata=lWRVCh0Tnan8ZQyNcC4vp%2BahO6nkEl51pOO6t
1Z1w50%3D&amp;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fapc
01.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Furldefens
e.com*2Fv3*2F__https*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo*
2Fxcat-user__*3B!!JFdNOqOXpB6UZW0!7CP7jiJu_SapsF7A5b-99TpS0lU01tKQJu5B
Hi7YpIvbH3ekgOyxSZXEftL6i0z5gH4nOA*24*26amp*3Bdata*3D04*7C01*7Cjjohnso
n2*40lenovo.com*7Cf83e67c87894408a53e708d925d63f3e*7C5c7d0b28bdf8410ca
a934df372b16203*7C1*7C0*7C637582424394899812*7CUnknown*7CTWFpbGZsb3d8e
yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C10
00*26amp*3Bsdata*3DMv26iG1vAmV6VkL9T2y9y7mCw2HynRccrgLEmkI7qPE*3D*26am
p*3Breserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6U
ZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfhpT0
Lceg*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b49d9
431173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C6375
82509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l
uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DW2RnPg6*2Fg
LMHOeUd*2BIWpAOcpR8L611KCet3RMfR9QSc*3D%26amp%3Breserved%3D0__%3BJSUlJ
SUlJSUlJSUqKioqKioqKioqKioqJSUlKioqKioqKioqKioqJSUlKiUlJSUlJSUlJSUlJSU
lJSUlJSUl!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4A
mC2ZGd1KGIY3HsiUf_MIx-aW5dA%24&amp;data=04%7C01%7Cjjohnson2%40lenovo.c
om%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b162
03%7C1%7C0%7C637583091425228336%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=
ta6B5GOnTfihdGlDxWtSRSEXO93v5VltKvkqjexE0XE%3D&amp;reserved=0>>



_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Flis__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwT28Xg0FQ$
ts.sourceforge.net%2Flists%2Flistinfo%2Fxcat-user&amp;data=04%7C01%7C
jjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28b
df8410caa934df372b16203%7C1%7C0%7C637583091425228336%7CUnknown%7CTWFp
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
n0%3D%7C1000&amp;sdata=XUCmt%2F4MAP6cVf5TVLjDWbCRsEyB%2FJ%2FVsjfv4XVv
X%2FE%3D&amp;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo*
2Fxcat-user%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b
49d9431173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C
637582509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo
iV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3D0CSRpbq
*2By49yGyAuyvlRxvYaKdzwoSa0YD69vJAQNIk*3D%26amp%3Breserved%3D0__%3BJSU
lJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDF
xPGI_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MLo7YZbdg%24&amp;data=04%7C01%7Cjjohn
son2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410
caa934df372b16203%7C1%7C0%7C637583091425228336%7CUnknown%7CTWFpbGZsb3d
8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
1000&amp;sdata=3nup2jzmVhmFM5%2F7tCL26rN%2FWmZTjPqDhuypLjlydxA%3D&amp;
reserved=0>

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Fur__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRjRqQRmQ$
ldefense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outloo
k.com%2F%3Furl%3Dhttps*3A*2F*2Flists.sourceforge.net*2Flists*2Flistin
fo*2Fxcat-user%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7Cf83e
67c87894408a53e708d925d63f3e*7C5c7d0b28bdf8410caa934df372b16203*7C1*7
C0*7C637582424394899812*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3D
1*2BBbHF*2Fq9PcNucuoI32A1qlGVfsV2yKX*2Btz80loBUj8*3D%26amp%3Breserved
%3D0__%3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!JFdNOqOXpB6UZW0!6qpRG5NAI9z1
J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfi7F465LQ%24&amp;dat
a=04%7C01%7Cjjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f
%7C5c7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583091425228336%7CUn
known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
WwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=6ZyRGUneBgWbYfnl8ugB6zcv7apIyulnx6
BWR3q9UrI%3D&amp;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fapc
01.safelinks.protection.outlook.com*2F*3Furl*3Dhttps*3A*2F*2Flists.sou
rceforge.net*2Flists*2Flistinfo*2Fxcat-user*26amp*3Bdata*3D04*7C01*7Cj
johnson2*40lenovo.com*7Cf83e67c87894408a53e708d925d63f3e*7C5c7d0b28bdf
8410caa934df372b16203*7C1*7C0*7C637582424394899812*7CUnknown*7CTWFpbGZ
sb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3
D*7C1000*26amp*3Bsdata*3D1*2BBbHF*2Fq9PcNucuoI32A1qlGVfsV2yKX*2Btz80lo
BUj8*3D*26amp*3Breserved*3D0__*3BJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!JFdNO
qOXpB6UZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7j
LMfi7F465LQ*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c5
45b49d9431173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0
*7C637582509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DvOgd
DEZJ0BadJgRX9OtnaC7UaTmK7XVD*2BVtqbywlQaU*3D%26amp%3Breserved%3D0__%3B
JSUlJSUlJSUlJSUqKioqKiolJSUqKioqKioqKioqKiolJSUqKioqJSUlJSUlJSUlJSUlJS
UlJSUlJQ!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPGI_aaz3lCWw4Am
C2ZGd1KGIY3HsiUf_MKKN41AUA%24&amp;data=04%7C01%7Cjjohnson2%40lenovo.co
m%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa934df372b1620
3%7C1%7C0%7C637583091425228336%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjA
wMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=j
dnJgLP5g3ZiOPeHoi8IDed%2BfuFfxP5jCNo1PYrJWHQ%3D&amp;reserved=0>>



_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Flists.sourceforge.net%2Flists%2Fli
stinfo%2Fxcat-user__%3B!!JFdNOqOXpB6UZW0!6qpRG5NAI9z1J0vKAjYRLInh8G_C
e4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfiyL43Yzw%24&amp;data=04%7C01%7Cjjoh
nson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf84
10caa934df372b16203%7C1%7C0%7C637583091425238336%7CUnknown%7CTWFpbGZs
b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
D%7C1000&amp;sdata=nC3rylSs6lv8SqNXnkUI1A7i8XT2CAbHMF3KSTCfFRw%3D&amp
;reserved=0

<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Flis
ts.sourceforge.net*2Flists*2Flistinfo*2Fxcat-user__*3B!!JFdNOqOXpB6UZW
0!6qpRG5NAI9z1J0vKAjYRLInh8G_Ce4PMXR9wecE-uPtkM38P9Arf76U-o7jLMfiyL43Y
zw*24%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b49d943
1173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C637582
509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM
zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3DBSU*2F844zxzh
H*2BaGO*2BQwuobBKKpYpjuG*2FHX7bjzQ2BVU*3D%26amp%3Breserved%3D0__%3BJSU
lJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_
sJofG7wQUpTDFxPGI_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MIvAKvqXw%24&amp;data=04
%7C01%7Cjjohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c
7d0b28bdf8410caa934df372b16203%7C1%7C0%7C637583091425238336%7CUnknown%
7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
VCI6Mn0%3D%7C1000&amp;sdata=bGc2MGoKERF1CCoXuG48LgX5MjhMBn7TK1IShdTYKz
g%3D&amp;reserved=0>



_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Flist__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwQG4M9nNA$
s.sourceforge.net%2Flists%2Flistinfo%2Fxcat-user&amp;data=04%7C01%7Cjj
ohnson2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8
410caa934df372b16203%7C1%7C0%7C637583091425238336%7CUnknown%7CTWFpbGZs
b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
%7C1000&amp;sdata=4ITia%2FVqhudv5xDbief%2F%2BnlYmDkX6UqY7B1lBppHHp0%3D
&amp;reserved=0
<https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furl__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwRxH1NrEQ$
defense.com%2Fv3%2F__https%3A%2F%2Fapc01.safelinks.protection.outlook.
com%2F%3Furl%3Dhttps*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo*
2Fxcat-user%26amp%3Bdata%3D04*7C01*7Cjjohnson2*40lenovo.com*7C670c545b
49d9431173a608d925ea22b8*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C
637582509812063820*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjo
iV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000%26amp%3Bsdata%3D0CSRpbq
*2By49yGyAuyvlRxvYaKdzwoSa0YD69vJAQNIk*3D%26amp%3Breserved%3D0__%3BJSU
lJSUlJSUlJSUlJSUlJSUlJSU!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDF
xPGI_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MLo7YZbdg%24&amp;data=04%7C01%7Cjjohn
son2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410
caa934df372b16203%7C1%7C0%7C637583091425238336%7CUnknown%7CTWFpbGZsb3d
8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C
1000&amp;sdata=Ok61iK%2BYm0Mmgx1xg7dEqh4cGOkdxSx1T%2Fzn%2FlJuHTQ%3D&am
p;reserved=0>


_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Furld__;JSUl!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwROAXONyw$
efense.com%2Fv3%2F__https%3A%2F%2Flists.sourceforge.net%2Flists%2Flist
info%2Fxcat-user__%3B!!JFdNOqOXpB6UZW0!-Y0C_83SB3EOTw_sJofG7wQUpTDFxPG
I_aaz3lCWw4AmC2ZGd1KGIY3HsiUf_MLniW8iCQ%24&amp;data=04%7C01%7Cjjohnson
2%40lenovo.com%7C5111016cfe4c415c61bc08d926718d4f%7C5c7d0b28bdf8410caa
934df372b16203%7C1%7C0%7C637583091425238336%7CUnknown%7CTWFpbGZsb3d8ey
JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C100
0&amp;sdata=LLPMeGkTlKQfIs%2B5yeLWIvvBuZvez1fZfgYrfWb5B9g%3D&amp;reser
ved=0


_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://apc01.safelinks.protection.outlook.com/?url=https*3A*2F*2Flists.sourceforge.net*2Flists*2Flistinfo*2Fxcat-user&amp;data=04*7C01*7Cjjohnson2*40lenovo.com*7C5111016cfe4c415c61bc08d926718d4f*7C5c7d0b28bdf8410caa934df372b16203*7C1*7C0*7C637583091425238336*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&amp;sdata=4ITia*2FVqhudv5xDbief*2F*2BnlYmDkX6UqY7B1lBppHHp0*3D&amp;reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwR_H2jm5A$

_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://urldefense.com/v3/__https://lists.sourceforge.net/lists/listinfo/xcat-user__;!!JFdNOqOXpB6UZW0!6XUUnBKqBTubAc1NRdEuF2xwenI68T9Tdl46PtQ9sX-3JiGZlM4UBdHN2iqaHwTBGiq--Q$



_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user

Re: [xcat-user] [External] Re: xCAT 2.16.2 new xNBA issue

Reply via email to