Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Sashi_K]

Hi Killian,

We are working on the problem and try to fix the issues to the best.

Regards,
Sashi

-Original Message-
From: Kilian Cavalotti [mailto:kilian.cavalotti.w...@gmail.com] 
Sent: Thursday, July 27, 2017 11:03 PM
To: Ponnandi, Soorej <soorej_ponna...@dell.com>; K, Sashi <sash...@dell.com>
Cc: linux-poweredge-Lists <linux-powere...@lists.us.dell.com>
Subject: Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

Hi Dell,

Did you get any chance to look at that issue? Would be great to see a fix make 
its way to the next release...

Thanks!
--
Kilian
___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Kilian Cavalotti]

Hi Dell,

Did you get any chance to look at that issue? Would be great to see a
fix make its way to the next release...

Thanks!
-- 
Kilian

___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Kilian Cavalotti]

Hi Dell,

I actually found a very simple reproducer.
The issue is not with the kickstart environment per se, but with
memory allocation within DSU.

It turns out that the RHEL/CentOS kickstart environment sets up the
MALLOC_PERTURB_ [1],[2] environment variable. And surely enough, as
soon as you define it in a regular, non-kickstart environment, DSU
starts exhibiting all sorts of aberrant behavior. Which very likely
indicates that parts of DSU's code use memory before it's even
initialized, or mistakenly re-use freed memory.

Setting MALLOC_PERTURB_, the first effect is that collecting inventory
doesn't work in non-interactive mode:
-- 8< 
# export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))
# dsu -n
Dell System Update 1.4.2
Copyright (C) 2014 Dell Proprietary.
Verifying catalog installation ...
Installing catalog from repository ...
Fetching dsucatalog ...
Reading the catalog ...
Fetching invcol_M18HC_LN64_17.06.000.1171_A00 ...
Verifying inventory collector installation ...
Getting System Inventory ...
Could not get the inventory
Exiting DSU!
-- 8< 

For reference, here's the "dsu -i" output when MALLOC_PERTURB_ is undefined:
-- 8< 
[root@sh-101-59 ~]# unset MALLOC_PERTURB_
[root@sh-101-59 ~]# dsu -i
Dell System Update 1.4.2
Copyright (C) 2014 Dell Proprietary.
Verifying catalog installation ...
Installing catalog from repository ...
Fetching dsucatalog ...
Reading the catalog ...
Fetching invcol_M18HC_LN64_17.06.000.1171_A00 ...
Verifying inventory collector installation ...
Getting System Inventory ...
warning: Inventory collector returned with partial failure.

1. OpenManage Server Administrator  ( Version : 8.5.0 )

2. BIOS  ( Version : 2.4.2 )

3. Lifecycle Controller  ( Version : 2.41.40.40 )

4. Dell 32 Bit uEFI Diagnostics  ( Version : 4239A36 )

5. OS COLLECTOR 1.1  ( Version : OSC_1.1 )

6. System CPLD  ( Version : 1.0.0 )

7.  iDRAC  ( Version : 2.41.40.40 )

8. Intel(R) Ethernet 10G X520 LOM  ( Version : 17.5.10 )

9. Intel(R) Ethernet 10G X520 LOM  ( Version : 17.5.10 )

10. OpenManage | iDRAC Service Module  ( Version : 0 )

Exiting DSU!
-- 8< 

Ok, so let's try to generate the inventory by hand:
-- 8< 
# chmod +x /usr/libexec/dell_dup/invcol_M18HC_LN64_17.06.000.1171_A00
# /usr/libexec/dell_dup/invcol_M18HC_LN64_17.06.000.1171_A00
-outc=/dev/shm/inv.xml


Unsupported operating system. 
   Invalid inventory results.
Unsupported operating system. 
   Invalid inventory
results.
-- 8< 

Not minding the XML error output, we now have an inventory file that
we can pass to DSU, in *non-interactive* mode (-n flag). See how it
now thinks that it's been invoked in preview mode (-p):
-- 8< 
# export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))
# dsu --input-inventory-file=/dev/shm/inv.xml -n
Dell System Update 1.4.2
Copyright (C) 2014 Dell Proprietary.
Verifying catalog installation ...
Installing catalog from repository ...
Fetching dsucatalog ...
Reading the catalog ...
Determining Applicable Updates ...
- Update Preview -
# : Component : Version : Filename
--
1 : OpenManage | iDRAC Service Module : 2.5 :
Systems-Management_Application_DT30R_LN64_2.5_A00
NOTE: The preview option displays the components which can be updated
based on Catalog. This option does not update.
Run --inventory for checking the component status post DSU commit.
Exiting DSU!
-- 8< 

Clearly the memory range where the arguments are stored is not the one
that is being read.

Finally, to quote Ulrich Drepper on the use of MALLOC_PERTURB_:
> This technique can find hard to detect bugs. It is therefore suggested to 
> always use this flag (at least temporarily) when testing out code or a new 
> distribution.

I would put a lot of emphasis on the word "testing"...

Now that you have an easy way to reproduce the issue, I hope that
you'll take the time to run Valgrind on DSU and fix all the memory
allocation/free'ing issues in a reasonable timeframe. Thanks.


[1]: http://udrepper.livejournal.com/11429.html
[2]: http://jhrozek.livejournal.com/1755.html

Cheers,
-- 
Kilian

PS: Please don't suggest to undefine MALLOC_PERTURB_ to work around the issue.

___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Kilian Cavalotti]

Hi Dell,

Any update on this? Did you manage to reproduce it? Do you have a fix?
It's been 2 months. Should we submit a formal support request to our
Dell representative?

Thanks!
-- 
Kilian

___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Kilian Cavalotti]

Hi Spike,

Thanks a lot for sharing this, it's comforting to know that the issue
has been observed other places. But I have to say it's also a bit
unsettling to learn that it occurred *within* Dell and still had made
its way to Dell's customers...
And thanks for the much easier reproducer.

Soorej, could you please give Spike's example a try? Our kickstart
infrastructure is quite involved and tied to many site-specific things
that will make it quite difficult to export to try in a different
environment.

Also, I'm not sure what you mean by "creating the iso". We're not
creating any ISO, we're just deploying servers, and would like to be
able to upgrade their firmwares in a single pass while they're being
deployed. Hence our wish to use "dsu -n" in the %post section of a
kickstart file.

Cheers,
-- 
Kilian


On Tue, May 23, 2017 at 9:00 AM,  <spike_wh...@dell.com> wrote:
> Dell - Internal Use - Confidential
>
> Killian,
>
>
>
> I work in Dell I/T, so my team is a consumer of dsu – like you. I believe
> this is very much related to a similar ‘dsu –n’ situation my team has
> experienced.
>
>
>
> And it’s probably easier to reproduce than having to re-kickstart a server
> every time.
>
>
>
> We have implemented a “firstboot” utility in our /etc/rc.local code.  Like
> so:
>
> # firstboot utility
>
> for script in `ls /etc/rc.d/init.d/firstboot.d/*`
>
> do
>
> echo "Executing script at $script"
>
> nohup $script >> /var/log/firstboot.log 2>&1 && rm $script
>
> sleep 10
>
> done
>
>
>
> The intent is if you wish to execute something not now, but on next reboot –
> you drop a little scriptlet in /etc/rc.d/init.d/firstboot.d/ dir.  And it
> gets executed on next reboot.  And then deleted, so it gets executed only on
> next reboot.
>
>
>
> Mainly we use this in our initial post-build customizations.  For those 1-2
> specific activities that we can’t perform on the install media’s kernel.
>
>
>
> Anyway, we have noticed that if we drop down a “firstboot” scriptlet that
> does ‘dsu –n’, it doesn’t do any flashing.   Whereas, in a regular
> environment ‘dsu –n’ flashes fine.
>
>
>
> We never were able to identify why.  Whether it was due to not having a
> controlling TTY, or what.  Ultimately, we went another away.   So at this
> point, it’s a curiosity item only – I’d forgotten until you mentioned it.
>
>
>
> Spike White
>
> Dell I/T
>
>
>
>
>
> Date: Sat, 20 May 2017 22:46:22 -0700
>
> From: Kilian Cavalotti <kilian.cavalotti.w...@gmail.com>
>
> Subject: Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'
>
> To: Patrick Boutilier <bouti...@ednet.ns.ca>
>
> Cc: linux-poweredge@dell.com
>
> Message-ID:
>
>
> 

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Spike_White]

Dell - Internal Use - Confidential
Killian,

I work in Dell I/T, so my team is a consumer of dsu - like you. I believe this 
is very much related to a similar 'dsu -n' situation my team has experienced.

And it's probably easier to reproduce than having to re-kickstart a server 
every time.

We have implemented a "firstboot" utility in our /etc/rc.local code.  Like so:
# firstboot utility
for script in `ls /etc/rc.d/init.d/firstboot.d/*`
do
echo "Executing script at $script"
nohup $script >> /var/log/firstboot.log 2>&1 && rm $script
sleep 10
done

The intent is if you wish to execute something not now, but on next reboot - 
you drop a little scriptlet in /etc/rc.d/init.d/firstboot.d/ dir.  And it gets 
executed on next reboot.  And then deleted, so it gets executed only on next 
reboot.

Mainly we use this in our initial post-build customizations.  For those 1-2 
specific activities that we can't perform on the install media's kernel.

Anyway, we have noticed that if we drop down a "firstboot" scriptlet that does 
'dsu -n', it doesn't do any flashing.   Whereas, in a regular environment 'dsu 
-n' flashes fine.

We never were able to identify why.  Whether it was due to not having a 
controlling TTY, or what.  Ultimately, we went another away.   So at this 
point, it's a curiosity item only - I'd forgotten until you mentioned it.

Spike White
Dell I/T



Date: Sat, 20 May 2017 22:46:22 -0700

From: Kilian Cavalotti 
<kilian.cavalotti.w...@gmail.com<mailto:kilian.cavalotti.w...@gmail.com>>

Subject: Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

To: Patrick Boutilier <bouti...@ednet.ns.ca<mailto:bouti...@ednet.ns.ca>>

Cc: linux-poweredge@dell.com<mailto:linux-poweredge@dell.com>

Message-ID:

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Soorej_Ponnandi]

Dell - Internal Use - Confidential  

Kilian,

As per your mail:
> It's called as "/sbin/dsu -n" in the %post section of my kickstart 
> file, and yet runs as it had been called as "dsu -p"

As I understand, the commands in the %post section will be executed while 
creating the iso (in chroot, nochroot environment), but not while booting a 
platform using the iso. Is it intentional to use dsu -n in post section?

Also, could you share the kickstart file you used for us to try.

Thanks,
Soorej

-Original Message-
From: linux-poweredge-bounces-Lists On Behalf Of K, Sashi
Sent: Monday, May 22, 2017 10:33 PM
To: linux-poweredge-Lists <linux-powere...@lists.us.dell.com>
Subject: Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

Hi,

We are looking in to this issue and will get back at the earliest.

Regards,
Sashi
sash...@dell.com 


-Original Message-
From: linux-poweredge-bounces-Lists On Behalf Of linux-poweredge-request-Lists
Sent: Sunday, May 21, 2017 10:30 PM
To: linux-poweredge-Lists <linux-powere...@lists.us.dell.com>
Subject: Linux-PowerEdge Digest, Vol 156, Issue 14

Send Linux-PowerEdge mailing list submissions to
linux-poweredge@dell.com

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.us.dell.com/mailman/listinfo/linux-poweredge
or, via email, send a message with subject or body 'help' to
linux-poweredge-requ...@dell.com

You can reach the person managing the list at
linux-poweredge-ow...@dell.com

When replying, please edit your Subject line so it is more specific than "Re: 
Contents of Linux-PowerEdge digest..."


Today's Topics:

   1.  DSU replaces '-n' option with '-p' (Kilian Cavalotti)
   2. Re:  DSU replaces '-n' option with '-p' (Patrick Boutilier)
   3. Re:  DSU replaces '-n' option with '-p' (Kilian Cavalotti)


--

Message: 1
Date: Sat, 20 May 2017 17:45:48 -0700
From: Kilian Cavalotti <kilian.cavalotti.w...@gmail.com>
Subject: [Linux-PowerEdge] DSU replaces '-n' option with '-p'
To: linux-powere...@lists.us.dell.com
Message-ID:

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Sashi_K]

Hi,

We are looking in to this issue and will get back at the earliest.

Regards,
Sashi
sash...@dell.com 


-Original Message-
From: linux-poweredge-bounces-Lists On Behalf Of linux-poweredge-request-Lists
Sent: Sunday, May 21, 2017 10:30 PM
To: linux-poweredge-Lists <linux-powere...@lists.us.dell.com>
Subject: Linux-PowerEdge Digest, Vol 156, Issue 14

Send Linux-PowerEdge mailing list submissions to
linux-poweredge@dell.com

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.us.dell.com/mailman/listinfo/linux-poweredge
or, via email, send a message with subject or body 'help' to
linux-poweredge-requ...@dell.com

You can reach the person managing the list at
linux-poweredge-ow...@dell.com

When replying, please edit your Subject line so it is more specific than "Re: 
Contents of Linux-PowerEdge digest..."


Today's Topics:

   1.  DSU replaces '-n' option with '-p' (Kilian Cavalotti)
   2. Re:  DSU replaces '-n' option with '-p' (Patrick Boutilier)
   3. Re:  DSU replaces '-n' option with '-p' (Kilian Cavalotti)


--

Message: 1
Date: Sat, 20 May 2017 17:45:48 -0700
From: Kilian Cavalotti <kilian.cavalotti.w...@gmail.com>
Subject: [Linux-PowerEdge] DSU replaces '-n' option with '-p'
To: linux-powere...@lists.us.dell.com
Message-ID:

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Kilian Cavalotti]

Thanks for the suggestion. "dsu -n" actually applies the updates when run
in a regular environment, outside of the Kickstart file.

Cheers,
-- 
Kilian

On May 20, 2017 5:57 PM, "Patrick Boutilier"  wrote:

> On 05/20/2017 09:45 PM, Kilian Cavalotti wrote:
>
>> HI,
>>
>> I'm trying to run an unattended upgrade of firmware as part of my
>> servers deployment process and I have all the difficulties in the work
>> to make that work.
>>
>> One of puzzling issues is that in the deployment environment (regular
>> Kickstarted Anaconda, deploying CentOS servers), "dsu -n" somehow
>> manages to think it's called in "preview" mode. The extremely
>> frustrating outcome is that it lists the updates to apply, but just
>> skips them.
>>
>> Here's the log:
>>
>> [63253]: + /sbin/dsu -n
>> [63253]: Dell System Update 1.4.2
>> [63253]: Copyright (C) 2014 Dell Proprietary.
>> [63253]: Verifying catalog installation ...
>> [63253]: Installing catalog from repository ...
>> [63253]: Fetching dsucatalog ...
>> [63253]: Reading the catalog ...
>> [63253]: Determining Applicable Updates ...
>> [63253]: - Update Preview -
>> [63253]: # : Component : Version : Filename
>> [63253]: --
>> [63253]: 1 : OS COLLECTOR 2.1, A00 : OSC_1.1 :
>> Diagnostics_Application_5D53Y_LN64_OSC_1.1_A00
>> [63253]: 2 :  iDRAC : 2.41.40.40 :
>> iDRAC-with-Lifecycle-Controller_Firmware_4950Y_LN_2.41.40.40_A00
>> [63253]: 3 : Intel(R) Ethernet 10G X520 LOM : 17.5.10 :
>> Network_Firmware_V6TPJ_LN64_17.5.10_A00
>> [63253]: NOTE: The preview option displays the components which can be
>> updated based on Catalog. This option does not update.
>> [63253]: Run --inventory for checking the component status post DSU
>> commit.
>> [63253]: Exiting DSU!
>>
>> It's called as "/sbin/dsu -n" in the %post section of my kickstart
>> file, and yet runs as it had been called as "dsu -p"
>>
>> Does anybody has any clue why this is happening? Could Dell please
>> provide a fix?
>>
>
> Maybe you need a "-u" as well to tell dsu to do the update? I don't have
> any boxes that need any firmware updates so I can't test.
>
>
>
>> Thanks,
>>
>>
>
> ___
> Linux-PowerEdge mailing list
> Linux-PowerEdge@dell.com
> https://lists.us.dell.com/mailman/listinfo/linux-poweredge
>
>
___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

Re: [Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Patrick Boutilier]

On 05/20/2017 09:45 PM, Kilian Cavalotti wrote:

HI,

I'm trying to run an unattended upgrade of firmware as part of my
servers deployment process and I have all the difficulties in the work
to make that work.

One of puzzling issues is that in the deployment environment (regular
Kickstarted Anaconda, deploying CentOS servers), "dsu -n" somehow
manages to think it's called in "preview" mode. The extremely
frustrating outcome is that it lists the updates to apply, but just
skips them.

Here's the log:

[63253]: + /sbin/dsu -n
[63253]: Dell System Update 1.4.2
[63253]: Copyright (C) 2014 Dell Proprietary.
[63253]: Verifying catalog installation ...
[63253]: Installing catalog from repository ...
[63253]: Fetching dsucatalog ...
[63253]: Reading the catalog ...
[63253]: Determining Applicable Updates ...
[63253]: - Update Preview -
[63253]: # : Component : Version : Filename
[63253]: --
[63253]: 1 : OS COLLECTOR 2.1, A00 : OSC_1.1 :
Diagnostics_Application_5D53Y_LN64_OSC_1.1_A00
[63253]: 2 :  iDRAC : 2.41.40.40 :
iDRAC-with-Lifecycle-Controller_Firmware_4950Y_LN_2.41.40.40_A00
[63253]: 3 : Intel(R) Ethernet 10G X520 LOM : 17.5.10 :
Network_Firmware_V6TPJ_LN64_17.5.10_A00
[63253]: NOTE: The preview option displays the components which can be
updated based on Catalog. This option does not update.
[63253]: Run --inventory for checking the component status post DSU commit.
[63253]: Exiting DSU!

It's called as "/sbin/dsu -n" in the %post section of my kickstart
file, and yet runs as it had been called as "dsu -p"

Does anybody has any clue why this is happening? Could Dell please
provide a fix?


Maybe you need a "-u" as well to tell dsu to do the update? I don't have 
any boxes that need any firmware updates so I can't test.





Thanks,



<>___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge

[Linux-PowerEdge] DSU replaces '-n' option with '-p'

[Kilian Cavalotti]

HI,

I'm trying to run an unattended upgrade of firmware as part of my
servers deployment process and I have all the difficulties in the work
to make that work.

One of puzzling issues is that in the deployment environment (regular
Kickstarted Anaconda, deploying CentOS servers), "dsu -n" somehow
manages to think it's called in "preview" mode. The extremely
frustrating outcome is that it lists the updates to apply, but just
skips them.

Here's the log:

[63253]: + /sbin/dsu -n
[63253]: Dell System Update 1.4.2
[63253]: Copyright (C) 2014 Dell Proprietary.
[63253]: Verifying catalog installation ...
[63253]: Installing catalog from repository ...
[63253]: Fetching dsucatalog ...
[63253]: Reading the catalog ...
[63253]: Determining Applicable Updates ...
[63253]: - Update Preview -
[63253]: # : Component : Version : Filename
[63253]: --
[63253]: 1 : OS COLLECTOR 2.1, A00 : OSC_1.1 :
Diagnostics_Application_5D53Y_LN64_OSC_1.1_A00
[63253]: 2 :  iDRAC : 2.41.40.40 :
iDRAC-with-Lifecycle-Controller_Firmware_4950Y_LN_2.41.40.40_A00
[63253]: 3 : Intel(R) Ethernet 10G X520 LOM : 17.5.10 :
Network_Firmware_V6TPJ_LN64_17.5.10_A00
[63253]: NOTE: The preview option displays the components which can be
updated based on Catalog. This option does not update.
[63253]: Run --inventory for checking the component status post DSU commit.
[63253]: Exiting DSU!

It's called as "/sbin/dsu -n" in the %post section of my kickstart
file, and yet runs as it had been called as "dsu -p"

Does anybody has any clue why this is happening? Could Dell please
provide a fix?

Thanks,
-- 
Kilian

___
Linux-PowerEdge mailing list
Linux-PowerEdge@dell.com
https://lists.us.dell.com/mailman/listinfo/linux-poweredge