[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Scott Moser]

This bug is believed to be fixed in curtin in 17.1. If this is still a
problem for you, please make a comment and set the state back to New

Thank you.

** Changed in: curtin
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Paolo de Rosa]

** Attachment added: "debug_md_device.txt"
   
https://bugs.launchpad.net/curtin/+bug/1618429/+attachment/4809253/+files/debug_md_device.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ryan Harper]

If we can get the full installation log with the same error, please open
up a new bug against curtin.  While the error mentioned in comment #30
is mdadm related, this bug addressed errors during mdadm scan of
previous disks and the issue in comment #30 is something else.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ryan Harper]

On Wed, Jan 18, 2017 at 8:43 AM, Ante Karamatić <
ante.karama...@canonical.com> wrote:

> Yes, failures are consistent and happen even after 'erase disks' is enabled
> in MAAS. They also happen on all nodes. I'm afraid I can't get any more
> logs than this. This is all that MAAS can capture in this environment.
>


Hrm, it would be *really* helpful to have the complete install log.  In
particular, curtin dumps
out the storage relationship of the disks before attempting to apply the
storage config

For example, when I attempt to recreate this by redeploying the same config
to the same disks
I can see this output:

Current device storage tree:
vdc
|-- vdc1
|-- vdc2
|   `-- md0
`-- vdc3
`-- md1
vdd
|-- vdd1
|-- vdd2
|   `-- md0
`-- vdd3
`-- md1

Which shows the relationship that was discovered and cleared.  We then clear
the tree from the bottom up; wiping md superblocks and metadata,
wiping partitions, and then devices.  Resulting in a successful clearing
out of
previous on-disk storage configuration.  And install succeeds.  Something
is different between these systems and lack of debug info is going to
prevent
us from coming to a resolution.

The full log details the steps and the lists of current block device
holders.


If we have access to a node, then I'd like to get a node in failed state to
inspect.

https://gist.github.com/smoser/2610e9b78b8d7b54319675d9e3986a1b

That has details on how to stop a deployment node:

ssh ubuntu@$HOST_IP sudo touch /run/block-curtin-poweroff

If you can, then after a failed curtin install, getting at
/var/log/curtin/install.log
will contain everything we need to investigate further.


> On Wed, Jan 18, 2017 at 3:00 PM Ryan Harper <1618...@bugs.launchpad.net>
> wrote:
>
> > Is it possible to include the entire installation.log?
> >
> > Do repeat deploys to the same node and same configuration fail
> > consistently?
> >
> > On Wed, Jan 18, 2017 at 6:50 AM, Ante Karamatić <
> > ante.karama...@canonical.com> wrote:
> >
> > > ** Attachment added: "installation.log"
> > >https://bugs.launchpad.net/curtin/+bug/1618429/+
> > > attachment/4805810/+files/installation.log
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1618429
> > >
> > > Title:
> > >   Curtin doesn't clean up previous MD configuration
> > >
> > > To manage notifications about this bug go to:
> > > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> > >
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> >   Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
> --
> Ante Karamatić
> ante.karama...@canonical.com
> Canonical
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
>   Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ante Karamatić]

Yes, failures are consistent and happen even after 'erase disks' is enabled
in MAAS. They also happen on all nodes. I'm afraid I can't get any more
logs than this. This is all that MAAS can capture in this environment.

On Wed, Jan 18, 2017 at 3:00 PM Ryan Harper <1618...@bugs.launchpad.net>
wrote:

> Is it possible to include the entire installation.log?
>
> Do repeat deploys to the same node and same configuration fail
> consistently?
>
> On Wed, Jan 18, 2017 at 6:50 AM, Ante Karamatić <
> ante.karama...@canonical.com> wrote:
>
> > ** Attachment added: "installation.log"
> >https://bugs.launchpad.net/curtin/+bug/1618429/+
> > attachment/4805810/+files/installation.log
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1618429
> >
> > Title:
> >   Curtin doesn't clean up previous MD configuration
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
>   Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>
-- 
Ante Karamatić
ante.karama...@canonical.com
Canonical

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Re: [Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ryan Harper]

Is it possible to include the entire installation.log?

Do repeat deploys to the same node and same configuration fail
consistently?

On Wed, Jan 18, 2017 at 6:50 AM, Ante Karamatić <
ante.karama...@canonical.com> wrote:

> ** Attachment added: "installation.log"
>https://bugs.launchpad.net/curtin/+bug/1618429/+
> attachment/4805810/+files/installation.log
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1618429
>
> Title:
>   Curtin doesn't clean up previous MD configuration
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ante Karamatić]

** Attachment added: "installation.log"
   
https://bugs.launchpad.net/curtin/+bug/1618429/+attachment/4805810/+files/installation.log

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ante Karamatić]

I'm not sure this is fixed. We are still seeing problems:

MAAS' curtin version is 0.1.0~bzr425-0ubuntu1~16.04.1. Installation log
and curtin-config (obfuscated) is provided as attachments.

** Attachment added: "curtin-config.txt"
   
https://bugs.launchpad.net/curtin/+bug/1618429/+attachment/4805809/+files/curtin-config.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Launchpad Bug Tracker]

This bug was fixed in the package curtin - 0.1.0~bzr425-0ubuntu1~16.04.1

---
curtin (0.1.0~bzr425-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  [ Scott Moser ]
  * debian/new-upstream-snapshot: add writing of debian changelog entries.

  [ Ryan Harper ]
  * New upstream snapshot.
- unittest,tox.ini: catch and fix issue with trusty-level mock of open
- block/mdadm: add option to ignore mdadm_assemble errors  (LP: #1618429)
- curtin/doc: overhaul curtin documentation for readthedocs.org
  (LP: #1351085)
- curtin.util: re-add support for RunInChroot  (LP: #1617375)
- curtin/net: overhaul of eni rendering to handle mixed ipv4/ipv6 configs
- curtin.block: refactor clear_holders logic into block.clear_holders and
  cli cmd
- curtin.apply_net should exit non-zero upon exception.  (LP: #1615780)
- apt: fix bug in disable_suites if sources.list line is blank.
- vmtests: disable Wily in vmtests
- Fix the unittests for test_apt_source.
- get CURTIN_VMTEST_PARALLEL shown correctly in jenkins-runner output
- fix vmtest check_file_strippedline to strip lines before comparing
- fix whitespace damage in tests/vmtests/__init__.py
- fix dpkg-reconfigure when debconf_selections was provided.
  (LP: #1609614)
- fix apt tests on non-intel arch
- Add apt features to curtin.  (LP: #1574113)
- vmtest: easier use of parallel and controlling timeouts
- mkfs.vfat: add force flag for formating whole disks  (LP: #1597923)
- block.mkfs: fix sectorsize flag  (LP: #1597522)
- block_meta: cleanup use of sys_block_path and handle cciss knames
  (LP: #1562249)
- block.get_blockdev_sector_size: handle _lsblock multi result return
  (LP: #1598310)
- util: add target (chroot) support to subp, add target_path helper.
- block_meta: fallback to parted if blkid does not produce output
  (LP: #1524031)
- commands.block_wipe:  correct default wipe mode to 'superblock'
- tox.ini: run coverage normally rather than separately
- move uefi boot knowledge from launch and vmtest to xkvm

 -- Ryan Harper   Mon, 03 Oct 2016 13:43:54
-0500

** Changed in: curtin (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Jon Grimm]

** Tags removed: verification-needed
** Tags added: verification-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Ryan Harper]

I'd like to add some comments for the SRU verification.

This issue was discovered on a customer's system which is not currently
available for verification.  Analyzing the error code in the original
bug was when mdadm assemble was called on an array that was incomplete
(some members not available) and produced a return code of '2'.  The
curtin code that handles mdadm assembly was not expecting a return code
of '2' and exited.

After examination of the process, curtin is attempting to assemble any
arrays to discover any block device dependencies in an effort to wipe
them all clean (ensuring that we can install and reboot into a system
configured as expected).  We determined that during storage preparation
curtin does not care specifically if assembly fails as it was going to
*wipe* the target block devices in any case; thus adding a flag to
ignore return codes from mdadm assembly during this phase was the
recommended solution.

We were not able to simulate this particular failure, so instead we have
a unittest which explicitly returns a value of 2 to simulate the
situation and ensure that during our prepare phase we ignore those
errors.

Despite not recreating this specific error, we feel confident that the
fix for wiping previous md devices is fixed in this release.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Jon Grimm]

Added José Pekkarinen, in hopes able to verify the xenial-proposed fix
for this bug.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1618429] Re: Curtin doesn't clean up previous MD configuration

[Andy Whitcroft]

** Also affects: curtin (Ubuntu)
   Importance: Undecided
   Status: New

** Also affects: curtin (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: curtin (Ubuntu)
   Status: New => Fix Released

** Changed in: curtin (Ubuntu Xenial)
   Status: New => Fix Committed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1618429

Title:
  Curtin doesn't clean up previous MD configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/curtin/+bug/1618429/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs