The Cloud SQL Proxy logs suggest to me that it may not be using the right credentials? It is possible that it is trying to use the cluster's "default service account"?
If you use "kubectl exec ... -ti /bin/sh" you should be able to examine the contents of the credentials file that is being passed to ensure that the contents match what you expect. You can also try to interactively start /cloud_sql_proxy with different flags to see if you can get it to start and print the "expected" log message. Good luck I hope that helps! On Thursday, June 22, 2017 at 9:46:52 AM UTC-4, Traiano Welcome wrote: > > Hi Evan > > > On Thu, Jun 22, 2017 at 5:34 PM, Evan Jones <evan....@triggermail.io > <javascript:>> wrote: > >> I know nothing about wordpress, but for what it is worth, we are using >> this Cloud SQL Proxy container with success. A few notes about the config >> you posted: >> >> >> * I'm assuming that where you have >> "-instances=[INSTANCE_CONNECTION_NAME]=tcp:[PORT]" you've replaced this >> with your Cloud SQL instance name (project:region:cloud_sql_name), and port >> with 3306, right? >> > > > I have this: > > command: ["/cloud_sql_proxy", "--dir=/cloudsql", > > "-instances=lol-staging:europe-west2:lol-staging-001=tcp:3306", > "-credential_file=/secrets/cloudsql/credentials.json"] > > I've redeployed and can see logs now, the wordpress container is failing > because it's failing connection to SQL via the proxy: > > > MySQL Connection Error: (2006) MySQL server has gone away > Warning: mysqli::mysqli(): MySQL server has gone away in - on line 10 > Warning: mysqli::mysqli(): Error while reading greeting packet. > PID=167 in - on line 10 > Warning: mysqli::mysqli(): (HY000/2006): MySQL server has gone away > in - on line 10 > > The cloud sql proxy container complains about an account not having > correct permissions (however I have given it editor permissions): > > kubectl logs pods/wordpress-2608214628-9bc9b cloudsql-proxy > > > 2017/06/22 13:40:15 Throttling > refreshCfg(lol-staging:europe-west2:lol-staging-001): it was only called > 24.005689746s ago > 2017/06/22 13:40:15 couldn't connect to > "lol-staging:europe-west2:lol-staging-001": ensure that the account has > access to "lol-staging:europe-west2:lol-staging-001" (and make sure there's > no typo in that name). Error during createEphemeral for > lol-staging:europe-west2:lol-staging-001: googleapi: Error 403: The client > is not authorized to make this request., notAuthorized > 2017/06/22 13:40:18 New connection for > "lol-staging:europe-west2:lol-staging-001" > > > So I'm now wondering how to better validate which account is being > referred to here and whether the permissions are correct. > > > > > > > > > > > > > > > > > > > > >> >> * If you run kubectl logs pod cloudsql-proxy you should see something >> like the following: >> >> 2017/06/21 20:45:06 Listening on (whatever) for (your instance name) >> 2017/06/21 20:45:06 Ready for new connections >> >> * If the cloudsql-proxy container is running, you can run kubectl exec >> (pod) -c cloudsql-proxy -ti /bin/sh to get a shell and try to verify the >> cloudsql-proxy configuration, or to start a new instance and make sure it >> is configured correctly. >> >> * We are using this with local Unix socket connections in the /cloudsql >> directory, but localhost sockets should also work. >> >> >> I'm surprised you don't see any logs from your wordpress container, but I >> can't help with that part. >> >> Good luck! >> >> >> On Thursday, June 22, 2017 at 3:15:26 AM UTC-4, Traiano Welcome wrote: >>> >>> Hi Ahmet >>> >>> On Thursday, 22 June 2017 02:25:05 UTC+4, Ahmet Alp Balkan wrote: >>>> >>>> Can you run "kubectl logs -l app=wordpress"? I am assuming there will >>>> be some logs from the crashing mysql container. >>>> >>>> >>> Thanks for your response. I get no output from running that command (I >>> suppose no logs are being generated). I tried both the command, and running >>> it in a loop, just in case: >>> >>> for i in `seq 1 100`;do echo "checking $i " - $(kubectl logs -l app=web >>> );done >>> >>> No result. >>> >>> Just to be clear, wordpress container is in a pod, and the cloud sql >>> container is a sidecar container connected to pod. >>> >>> Google's documentation seems pretty thin on getting this working >>> properly with GKE containers, and non-existent on troubleshooting >>> connectivity issues with cloud sql proxy. >>> >>> Do you know of an alternative method of getting a container access to >>> Cloud SQL from GKE ? >>> >>> >>> >>> On Wed, Jun 21, 2017 at 7:58 AM, Traiano Welcome <tra...@gmail.com> >>>> wrote: >>>> >>>>> I'm following this example of how to get wordpress running on GKE >>>>> connected to Google Cloud SQL via the Google Cloud SQL Proxy: >>>>> >>>>> >>>>> https://cloud.google.com/sql/docs/mysql/connect-container-engine >>>>> >>>>> >>>>> <https://cloud.google.com/sql/docs/mysql/connect-container-engine> >>>>> >>>>> Unfortunately, my wordpress pod is failing with a crashloop error, and >>>>> it's not clear from the documentation how to dig deeper into the reason >>>>> for >>>>> this. Here is a sample of the error: >>>>> >>>>> >>>>> bash-3.2$ kubectl get pods| egrep wordpress >>>>> wordpress-713960421-v4f49 0/2 CrashLoopBackOff 16 20m >>>>> >>>>> (kubectl describe pod ...) >>>>> >>>>> 11m 22s 36 kubelet, gke-noon-staging-default- >>>>> pool-d500b601-dfb6 Warning FailedSync >>>>> Error syncing pod, skipping: [failed to "StartContainer" for "web" >>>>> with CrashLoopBackOff: "Back-off 5m0s restarting failed container=web >>>>> pod=wordpress-713960421-v4f49_default(f64276d2-5660-11e7-a565-42010a9a0023)" >>>>> , failed to "StartContainer" for "cloudsql-proxy" with >>>>> CrashLoopBackOff: "Back-off 5m0s restarting failed >>>>> container=cloudsql-proxy >>>>> pod=wordpress-713960421-v4f49_default(f64276d2-5660-11e7- >>>>> a565-42010a9a0023)" >>>>> ] >>>>> >>>>> My questions are: >>>>> >>>>> - How to dig further into why the pod is failing to deploy with >>>>> cloud sql proxy >>>>> - Has anyone got an example working for using cloud sql proxy with >>>>> as simple mysql client pod? >>>>> >>>>> Here is a description of the deployment (kubectl describe wordpress): >>>>> >>>>> >>>>> https://pastebin.com/DjYM97R7 <https://pastebin.com/DjYM97R7> >>>>> >>>>> >>>>> <https://pastebin.com/DjYM97R7> >>>>> >>>>> And a description of the pod (kubectl describe ): >>>>> >>>>> >>>>> <https://pastebin.com/pN7gUZg8> >>>>> >>>>> https://pastebin.com/pN7gUZg8 <https://pastebin.com/pN7gUZg8> >>>>> >>>>> >>>>> >>>>> I deployed the cloud sql proxy and wordpress container separately and >>>>> found the cloud sql proxy runs fine, but the wordpress blog container >>>>> fails >>>>> to startup on kubernetes. >>>>> Error syncing pod, skipping: >>>>> >>>>> failed to "StartContainer" for "wordpress" with CrashLoopBackOff >>>>> >>>>> <https://stackoverflow.com/questions/44672411/simple-client-connecting-to-google-cloud-sql-using-cloud-sql-proxy#comment76333823_44672411> >>>>> >>>>> Looking at the pod logs for wordpress, it seems wordpress is dying >>>>> because it cannot connect to the MySQL DB: >>>>> >>>>> MySQL Connection Error: (2002) Connection refused – Traiano Welcome >>>>> <https://stackoverflow.com/users/6052496/traiano-welcome> 3 hours ago >>>>> <https://stackoverflow.com/questions/44672411/simple-client-connecting-to-google-cloud-sql-using-cloud-sql-proxy#comment76334135_44672411> >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Kubernetes user discussion and Q&A" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to kubernetes-use...@googlegroups.com. >>>>> To post to this group, send email to kubernet...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/kubernetes-users. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Kubernetes user discussion and Q&A" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/kubernetes-users/2kqHNT1Nbtc/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> kubernetes-use...@googlegroups.com <javascript:>. >> To post to this group, send email to kubernet...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/kubernetes-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
