Hello Cory- Also, what versions of CoreOS is this happening under? `cat /etc/os-release`.
Brandon On Sat, Aug 13, 2016 at 7:26 PM Brandon Philips <[email protected]> wrote: > Hello Cory- > > There are a number of errors about the mounts which should be non-fatal: > > Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: > time="2016-08-09T15:40:08.520711349Z" level=error msg="Clean up Error! > Cannot destroy container > 26537dcd69eeb25dcd3a4867f81db909e5ca992efa1e6861d9e804770ec11239: > rmdriverfs: Driver overlay failed to remove root filesystem > 26537dcd69eeb25dcd3a4867f81db909e5ca992efa1e6861d9e804770ec11239: mount > still active" > Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: > time="2016-08-09T15:40:08.520749851Z" level=error msg="Handler for POST > /v1.22/containers/create returned error: Error relabeling upper directory: > operation not supported" > > Could you grab the output of `mount` from these machines? > > And it doesn't look like the daemon dies for several minutes after: > > Aug 09 15:43:41 ip-172-31-16-118.ec2.internal systemd[1]: Stopping Docker > Application Container Engine... > > I really don't know what is getting messed up to cause this. What are the > arguments to docker run? > > It looks sort of a like a repeat of > https://github.com/coreos/bugs/issues/1301 which was about SELinux and > OverlayFS so I am going to cc in Nick, Matthew, and Crawford. > > Cheers, > > Brandon > > On Tue, Aug 9, 2016 at 10:41 AM Cory Keane <[email protected]> wrote: > >> Hey Brandon, >> >> I decided to just email the logs directly to you. >> >> Thanks! >> >> On Tue, Aug 9, 2016 at 12:37 PM, Brandon Philips < >> [email protected]> wrote: >> >>> journalctl --no-pager -u docker.service > foo.log >>> >>> I think it turns on no-pager by default if it knows stdout isn't a tty >>> >>> On Tue, Aug 9, 2016 at 10:35 AM Cory Keane <[email protected]> wrote: >>> >>>> No worries Brandon. I will upload them. But what's the easiest way to >>>> export them? The command you told me to run makes it so I have to press the >>>> space bar to cycle through all the lines of the file. >>>> >>>> Thanks so much. >>>> >>>> On Tue, Aug 9, 2016 at 12:34 PM, Brandon Philips < >>>> [email protected]> wrote: >>>> >>>>> Hey Cory- >>>>> >>>>> It is really hard to know what to look for in the logs. If you don't >>>>> want to share the logs publicly you can email them to me personally and I >>>>> will take a look. >>>>> >>>>> Cheers, >>>>> >>>>> Brandon >>>>> >>>>> On Tue, Aug 9, 2016 at 8:42 AM Cory Keane <[email protected]> wrote: >>>>> >>>>>> Brandon, I apologize. It appears that this worked for one of the >>>>>> machines experiencing this issue, but not the other machine. Since I >>>>>> posted >>>>>> this, another one of our fleet machines auto updated to what appears to >>>>>> be >>>>>> this faulty version. >>>>>> >>>>>> So please let me know what I am looking for specifically when running >>>>>> the `journalctl -u docker` command. >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> *Cory Keane* >>>> Co-Founder & CTO >>>> ThreadMeUp.com >>>> >>> >> >> >> -- >> *Cory Keane* >> Co-Founder & CTO >> ThreadMeUp.com >> >
