Hello, We have run into an issue running Percona with O_DIRECT when using boot2docker (which uses aufs). We have been able to work around this problem by forcing docker not to use aufs in this case, but thought you might be interested in case there is an underlying problem that needs to be resolved. * Observed behaviour: - The mysql process becoming unresponsive and all attempts to access (e.g. ls) /var/lib/mysql (the mysql data directory) hang indefinitely. - Attempts to access /var/lib/mysql inside the container, and also outside the container (navigating to the associated directory in /var/lib/docker) both hang. * Important info: - This percona option must be set for the issue to occur: innodb_flush_method = O_DIRECT - This occurs when running Percona mysql in a docker container with using versions of boot2docker > 1.8.2, but versions <= 1.8.2 do not have the issue. - The boot2docker commit that introduced this issue is this commit where the kernel and aufs versions are bumped: [1]https://github.com/boot2docker/boot2docker/commit/7fb8d2d010e268 2512b82c7cebd0f743d61b44aa - The problem does not occur if /var/lib/mysql is declared as a VOLUME when building the docker image (docker will create a volume on the host, not using aufs) - The problem does not occur if a directory on the host is mounted into /var/lib/mysql It looks like docker has explicitly added support for direct IO with aufs by default ([2]https://github.com/docker/docker/pull/10534) so it seems to me like the assumption is that this use case should work (even if the best practice is to declare /var/lib/mysql as a VOLUME). * Steps to reproduce: # Using latest boot2docker ([3]https://github.com/boot2docker/boot2docker/releases/tag/v1.10.2),orany version > 1.8.2, create a VM: docker-machine create --driver virtualbox --virtualbox-cpu-count "1" --virtualbox-memory "4096" repro # Build a percona image - with the O_DIRECT option set - using this VM: cat > ./my.cnf <<- EOM [client] socket = /var/lib/mysql/mysql.sock [mysqld_safe] socket = /var/lib/mysql/mysql.sock [mysqld] user = mysql pid-file = /var/lib/mysql/mysql.pid socket = /var/lib/mysql/mysql.sock datadir = /var/lib/mysql bind-address = 0.0.0.0 innodb_flush_method = O_DIRECT EOM cat > Dockerfile <<- EOM FROM centos:6.6 RUN yum install -y [4]http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona -release-0.1-3.noarch.rpm RUN yum install -y Percona-Server-server-55 ADD my.cnf /etc/my.cnf #VOLUME ["/var/lib/mysql"] ENTRYPOINT ["mysqld"] EXPOSE 3306 EOM eval "$(docker-machine env repro)" docker build -t percona -f Dockerfile . # Run a container using this image docker run -d --name test_db percona # In one terminal window, exec into the container and watch /var/lib/mysql eval "$(docker-machine env repro)" docker exec -it test_db bash -c "TERM='xterm' watch ls -la /var/lib/mysql" # In another terminal window, run this command to attempt to create 20k tables in this container eval "$(docker-machine env repro)" docker exec -it test_db bash -c 'mysql -u root -e "create database testing" && date && for i in {1..20000}; do echo "create table table_$i (ID bigint(20) unsigned NOT NULL AUTO_INCREMENT, time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, f1 varchar(16) COLLATE utf8_bin NOT NULL, f2 char(16) COLLATE utf8_bin NOT NULL,  f11 varchar(16) COLLATE utf8_bin NOT NULL,  f111 varchar(16) COLLATE utf8_bin NOT NULL,  f22 varchar(16) COLLATE utf8_bin NOT NULL,  f222 varchar(16) COLLATE utf8_bin NOT NULL, PRIMARY KEY (ID));" | mysql -u root testing; echo -n "table_$i "; done' # At some point, usually less than a minute later, the watch command will hang, and any attempts to access /var/lib/mysql in the container or on the host will hang * Other info requested This is the info the aufs github README requests be included in this email: - /proc/mounts (instead of the output of mount(8)) docker@repro:~$ cat /proc/mounts tmpfs / tmpfs rw,relatime,size=3641328k 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,relatime 0 0 devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0 tmpfs /dev/shm tmpfs rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 /dev/sda1 /mnt/sda1 ext4 rw,relatime,data=ordered 0 0 cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0 cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls cgroup rw,relatime,net_cls 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/net_prio cgroup rw,relatime,net_prio 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb 0 0 Users /Users vboxsf rw,nodev,relatime 0 0 /dev/sda1 /mnt/sda1/var/lib/docker/aufs ext4 rw,relatime,data=ordered 0 0 none /mnt/sda1/var/lib/docker/aufs/mnt/61519d5d301361fe9094a732a8e79168930311bbf7 8298deda27f94665547a9d aufs rw,relatime,si=82437bd0fc319d67,dio,dirperm1 0 0 shm /mnt/sda1/var/lib/docker/containers/065adec0e8ca3533546118cc0930bd28d0aaa213 12f8442d2d4e5c6c4bc2dc56/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0 nsfs /var/run/docker/netns/8fed2b224c57 nsfs rw 0 0 - /sys/module/aufs/* root@repro:/sys/module/aufs# for f in `find . -type f`; do echo "$f:"; cat "$f"; done ./parameters/brs: 1 ./parameters/allow_userns: N ./version: 4.1.13+-20160215 ./uevent: cat: can't open './uevent': Permission denied - /sys/fs/aufs/* (if you have them) root@repro:/sys/fs/aufs# for f in `find . -type f`; do echo "$f:"; cat "$f"; done ./si_82437bd0fc319d67/br0: /mnt/sda1/var/lib/docker/aufs/diff/61519d5d301361fe9094a732a8e79168930311bbf 78298deda27f94665547a9d=rw ./si_82437bd0fc319d67/br1: /mnt/sda1/var/lib/docker/aufs/diff/61519d5d301361fe9094a732a8e79168930311bbf 78298deda27f94665547a9d-init=ro+wh ./si_82437bd0fc319d67/br2: /mnt/sda1/var/lib/docker/aufs/diff/5e08725fc1388127a22c01345ba5d84dbb11511cb 75d616b49d2ec848fb79792=ro+wh ./si_82437bd0fc319d67/br3: /mnt/sda1/var/lib/docker/aufs/diff/86e34d747e3ae936effd10b0059b178d56d33ad7f f4fb70a32d550e7b83f66a6=ro+wh ./si_82437bd0fc319d67/br4: /mnt/sda1/var/lib/docker/aufs/diff/5cc807e6634443dec30c3c22ae5b04e97e3d13170 9df81d8cae67b86e4bb9bf1=ro+wh ./si_82437bd0fc319d67/br5: /mnt/sda1/var/lib/docker/aufs/diff/0d8442795212973eaae827f91c0c2547b0dc27a64 1ddc7da30435fe6539a8b0d=ro+wh ./si_82437bd0fc319d67/br6: /mnt/sda1/var/lib/docker/aufs/diff/59f9ed479d38bfada4bb7af9c64874df256370870 071a8c83098c9d7f4f4c2e5=ro+wh ./si_82437bd0fc319d67/xi_path: /dev/shm/aufs.xino ./si_82437bd0fc319d67/brid0: 16384 ./si_82437bd0fc319d67/brid1: 16385 ./si_82437bd0fc319d67/brid2: 16386 ./si_82437bd0fc319d67/brid3: 16387 ./si_82437bd0fc319d67/brid4: 16388 ./si_82437bd0fc319d67/brid5: 16389 ./si_82437bd0fc319d67/brid6: 16390 - /debug/aufs/* (if you have them) Does not exist - linux kernel version  if your kernel is not plain, for example modified by distributor,  the url where i can download its source is necessary too. root@repro:/sys/module/aufs# uname -a Linux repro 4.1.18-boot2docker #1 SMP Sat Feb 20 08:24:27 UTC 2016 x86_64 GNU/Linux [5]https://github.com/boot2docker/boot2docker/releases/tag/v1.10.2 - aufs version which was printed at loading the module or booting the  system, instead of the date you downloaded. This is the output of docker info: root@repro:/sys/module/aufs# docker info Containers: 1  Running: 1  Paused: 0  Stopped: 0 Images: 8 Server Version: 1.10.2 Storage Driver: aufs  Root Dir: /mnt/sda1/var/lib/docker/aufs  Backing Filesystem: extfs  Dirs: 7  Dirperm1 Supported: true Execution Driver: native-0.2 Logging Driver: json-file Plugins:  Volume: local  Network: bridge null host Kernel Version: 4.1.18-boot2docker Operating System: Boot2Docker 1.10.2 (TCL 6.4.1); master : 611be10 - Mon Feb 22 22:47:06 UTC 2016 OSType: linux Architecture: x86_64 CPUs: 1 Total Memory: 3.858 GiB Name: repro ID: G2RW:EP4U:STXW:MP7M:255S:GUY4:YOXH:ZOPR:5UPB:KN3V:TT7D:DWQ2 Debug mode (server): true  File Descriptors: 23  Goroutines: 48  System Time: 2016-03-07T23:22:46.938269001Z  EventsListeners: 0  Init SHA1:  Init Path: /usr/local/bin/docker  Docker Root Dir: /mnt/sda1/var/lib/docker Labels:  provider=virtualbox This version of boot2docker is built with aufs 4.1.13+ (commit 1aa8a143457147473edc97035a7f4b89f9d234ae). [6]https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b7 3346b59e7b2d7ed/Dockerfile#L29 - configuration (define/undefine CONFIG_AUFS_xxx) [7]https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b7 3346b59e7b2d7ed/kernel_config#L4017 CONFIG_AUFS_FS=y # CONFIG_AUFS_BRANCH_MAX_127 is not set # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set CONFIG_AUFS_BRANCH_MAX_32767=y CONFIG_AUFS_SBILIST=y # CONFIG_AUFS_HNOTIFY is not set # CONFIG_AUFS_EXPORT is not set CONFIG_AUFS_XATTR=y # CONFIG_AUFS_FHSM is not set # CONFIG_AUFS_RDU is not set # CONFIG_AUFS_SHWH is not set # CONFIG_AUFS_BR_RAMFS is not set # CONFIG_AUFS_BR_FUSE is not set CONFIG_AUFS_BR_HFSPLUS=y CONFIG_AUFS_BDEV_LOOP=y # CONFIG_AUFS_DEBUG is not set - kernel configuration or /proc/config.gz (if you have it) [8]https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b7 3346b59e7b2d7ed/kernel_config Thanks very much! -- Christie Wilson Software Engineer Demonware
References 1. https://github.com/boot2docker/boot2docker/commit/7fb8d2d010e2682512b82c7cebd0f743d61b44aa 2. https://github.com/docker/docker/pull/10534 3. https://github.com/boot2docker/boot2docker/releases/tag/v1.10.2 4. http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm 5. https://github.com/boot2docker/boot2docker/releases/tag/v1.10.2 6. https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b73346b59e7b2d7ed/Dockerfile#L29 7. https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b73346b59e7b2d7ed/kernel_config#L4017 8. https://github.com/boot2docker/boot2docker/blob/611be104ad81065b8d1cf71b73346b59e7b2d7ed/kernel_config
------------------------------------------------------------------------------ Transform Data into Opportunity. Accelerate data analysis in your applications with Intel Data Analytics Acceleration Library. Click to learn more. http://makebettercode.com/inteldaal-eval
