Hi Anil, Thank you for linking me to your compose file. I have used a few good ideas from there in my own. I appreciate it.
As for the issue, with a little help I was able to trace it back to my own firewall. All is well right now. Thanks to everyone that responded to me! You were all very helpful and I am grateful. Best, Anthony On Aug 17, 2021, at 11:34 AM, Anil Kulkarni <[email protected]<mailto:[email protected]>> wrote: You don't often get email from [email protected]<mailto:[email protected]>. Learn why this is important<http://aka.ms/LearnAboutSenderIdentification> Hi Anthony, Can you please share your docker compose file? The file in my repo works fine for me https://github.com/anilkulkarni87/airflow-docker It was getting stuck owing to the user permission and was resolved. On Tue, Aug 17, 2021 at 5:58 AM Anthony Joyce <[email protected]<mailto:[email protected]>> wrote: Thanks Jarek. The commands below can grab the IP from the hostname, but nc -zvvn definitely hangs when trying to connect to my host IP on the Postgres port. This helps me narrow it down a bit. Thank you for that suggestion! Anthony On Aug 16, 2021, at 1:56 PM, Jarek Potiuk <[email protected]<mailto:[email protected]>> wrote: Another thing to try - just run airflow image with `bash` parameter (`docker run -it <image> bash`) and see if you can connect to your host/port (there is `nc` tool installed in the production image). I believe when you try it you will hang on trying to connect to your database. I think there is one reason why it could fail - in case your host name is wrongly parsed from the URL (because it contains restricted characters) - this could happen. This is what the connectivity check does (this is where it seems to hang) - copied it with explanation on what is happening (gethosbyname will also work for IP addresses BTW). You might try to reproduce this. # Checks if it is possible to connect to the host using netcat. # # We want to avoid misleading messages and perform only forward lookup of the service IP address. # Netcat when run without -n performs both forward and reverse lookup and fails if the reverse # lookup name does not match the original name even if the host is reachable via IP. This happens # randomly with docker-compose in GitHub Actions. # Since we are not using reverse lookup elsewhere, we can perform forward lookup in python # And use the IP in NC and add '-n' switch to disable any DNS use. # Even if this message might be harmless, it might hide the real reason for the problem # Which is the long time needed to start some services, seeing this message might be totally misleading # when you try to analyse the problem, that's why it's best to avoid it, ip=$(python -c "import socket; print(socket.gethostbyname('${host}'))") nc -zvvn "${ip}" "${port}" J. On Mon, Aug 16, 2021 at 7:31 PM Anthony Joyce <[email protected]<mailto:[email protected]>> wrote: Thanks for the suggestions. Will do. And yes, I am using the official apache-airflow image. Anthony On Aug 16, 2021, at 1:29 PM, Daniel Standish <[email protected]<mailto:[email protected]>> wrote: Let me offer a suggestion. I think it's helpful to simplify things and isolate components. To that end what you could do is scrap docker for a moment. Create a python virtualenv in your host. Install airflow into this virtualenv. Set AIRFLOW__CORE__SQL_ALCHEMY_CONN as appropriate and run airflow in the virtualenv. Then you'll be able to confirm the issue is docker and not some other connectivity issue. Next move on to docker, perhaps without compose. One thing you could do is get a terminal in a running container and try to connect to this server e.g. using telnet or psql (you might have to install these). Then you'll figure out what connection string will actually work and it's just a matter of setting up the env vars correctly for airflow to connect in the right way. Also, are you using the official apache-airflow image? If not, I would recommend that. On Mon, Aug 16, 2021 at 10:22 AM Anthony Joyce <[email protected]<mailto:[email protected]>> wrote: Hi Daniel, Thanks for the suggestion! I’m new at Docker so apologies if some of the configs are incorrect. Still learning. I removed service Postgres since I am using my local db. Still stuck on airflow-init. There are no log entries that explain what it is doing or what it is waiting for, but obviously airflow-init is waiting for something. I’ve since restarted docker service, flushed iptables chains relating to Docker, restarted again. Still stuck. Anthony On Aug 16, 2021, at 1:11 PM, Daniel Standish <[email protected]<mailto:[email protected]>> wrote: If you are using an already existing postgres, then why are you running postgres service in compose? If that's not the database you are connecting to, then you should remove it from your compose file. (conversely if you are trying to use compose postgres, then we might have some configuration suggestions to offer) --- Separately, on mac compose, for the container to access host i think you might need to add svc.localhost or something like that... there are SO posts on this topic eg https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mac -- +48 660 796 129 -- Cheers, Anil Kulkarni https://anilkulkarni.com/
