jason-brian-anderson commented on issue #30554: URL: https://github.com/apache/airflow/issues/30554#issuecomment-1502031635
Thanks for the fast reply @potiuk , I agree this does not seem to be an Airflow issue. Your feedback did give me hints that made the solution obvious; but it doesn't seem that docker-in-docker host bind mounts are impossible on Linux after all, as described below: ## Root cause: Wsl2 does not explicitly allow host bind mounts for docker-in-docker usage. There is a docker desktop file sharing configuration that must be set. The problem is that this necessary configuration interface is not present in the Docker Desktop UI (at least on v4.18.0) Settings->Resources tab **unless** you turn off wsl2 engine support under Settings->General. But to do this, you must install Hyper-V under “Turn Windows Features on and off”. Once you have restarted in Hyper-V Docker Desktop mode, you will then have the ‘File Sharing’ tab within Settings->Resources where you can set your development directory and subdirectories for mounting. Once you do this you can stay in Hyper-V mode as long as you don’t need to use the GPU, which as of this writing Hyper-V does not support, but you **can** go back to WSL2 mode and the file sharing settings only available in hyper-V mode will stick. If you ever need to edit filesharing, you’ll have to switch back temporarily to Hyper-V mode. _Note that GPU training support is not currently available in Hyper-V, so plan to live permanently in WSL2_ ## Solution: Installed and switch docker temporarily over to hyper-v. Specifically, i implemented hyper-v in windows features I unchecked Use the WSL 2 based engine This allowed access to a ‘file sharing’ interface in Resources that was not previously available. In it, i put my dev directory and restarted the airflow app Root was able to access the host mounted file system I then re-checked ‘Use the WSL 2 based engine’ and root was still able to access the host mounted fs. solved. ## Details Some details not included above - Once hyperv was installed, rebooted, and wsl 2 engine was unchecked, and the host mounts became available, GPU support was no longer available. I found from here that it is not supported: https://learn.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/gpu-acceleration But, fortunately, i could just switch back to Use WSL 2 for engine and the root local host bind mounts *were still available* even though the file sharing menu was no longer available! So clearly Docker’s claim from https://docs.docker.com/desktop/settings/windows/#file-sharing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
