Hello Cory- There are a number of errors about the mounts which should be non-fatal:
Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: time="2016-08-09T15:40:08.520711349Z" level=error msg="Clean up Error! Cannot destroy container 26537dcd69eeb25dcd3a4867f81db909e5ca992efa1e6861d9e804770ec11239: rmdriverfs: Driver overlay failed to remove root filesystem 26537dcd69eeb25dcd3a4867f81db909e5ca992efa1e6861d9e804770ec11239: mount still active" Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: time="2016-08-09T15:40:08.520749851Z" level=error msg="Handler for POST /v1.22/containers/create returned error: Error relabeling upper directory: operation not supported" Could you grab the output of `mount` from these machines? And it doesn't look like the daemon dies for several minutes after: Aug 09 15:43:41 ip-172-31-16-118.ec2.internal systemd[1]: Stopping Docker Application Container Engine... I really don't know what is getting messed up to cause this. What are the arguments to docker run? It looks sort of a like a repeat of https://github.com/coreos/bugs/issues/1301 which was about SELinux and OverlayFS so I am going to cc in Nick, Matthew, and Crawford. Cheers, Brandon On Tue, Aug 9, 2016 at 10:41 AM Cory Keane <[email protected]> wrote: > Hey Brandon, > > I decided to just email the logs directly to you. > > Thanks! > > On Tue, Aug 9, 2016 at 12:37 PM, Brandon Philips < > [email protected]> wrote: > >> journalctl --no-pager -u docker.service > foo.log >> >> I think it turns on no-pager by default if it knows stdout isn't a tty >> >> On Tue, Aug 9, 2016 at 10:35 AM Cory Keane <[email protected]> wrote: >> >>> No worries Brandon. I will upload them. But what's the easiest way to >>> export them? The command you told me to run makes it so I have to press the >>> space bar to cycle through all the lines of the file. >>> >>> Thanks so much. >>> >>> On Tue, Aug 9, 2016 at 12:34 PM, Brandon Philips < >>> [email protected]> wrote: >>> >>>> Hey Cory- >>>> >>>> It is really hard to know what to look for in the logs. If you don't >>>> want to share the logs publicly you can email them to me personally and I >>>> will take a look. >>>> >>>> Cheers, >>>> >>>> Brandon >>>> >>>> On Tue, Aug 9, 2016 at 8:42 AM Cory Keane <[email protected]> wrote: >>>> >>>>> Brandon, I apologize. It appears that this worked for one of the >>>>> machines experiencing this issue, but not the other machine. Since I >>>>> posted >>>>> this, another one of our fleet machines auto updated to what appears to be >>>>> this faulty version. >>>>> >>>>> So please let me know what I am looking for specifically when running >>>>> the `journalctl -u docker` command. >>>>> >>>>> >>> >>> >>> -- >>> *Cory Keane* >>> Co-Founder & CTO >>> ThreadMeUp.com >>> >> > > > -- > *Cory Keane* > Co-Founder & CTO > ThreadMeUp.com >
