Re: [lxc-users] AWS EC2: timeout connecting to instance metadata webserver (169.254.169.254) for *some* URLs (when connecting from a LXD container)
On 2020-11-19 00:07, Tomasz Chmielewski wrote: On 2020-11-18 23:50, Tomasz Chmielewski wrote: That's a weird one! In AWS, there is a concept of "instance metadata" - a webserver which lets you fetch some instance metadata using http: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html For example, you can run this (from both AWS/EC2 instance and LXD container running inside a AWS/EC2 instance), it will return some metadata: curl -v http://169.254.169.254/latest/meta-data/ Now, some of these requests time out when executed from a LXD container running inside a AWS/EC2 - but work perfectly from the very same AWS/EC2 instance. For example, this request works fine from AWS/EC2 instance (ignore the output - HTTP connection works just fine): root@aws-instance:~# curl -v http://169.254.169.254/latest/api/token * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) GET /latest/api/token HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/7.58.0 Accept: */* < HTTP/1.1 405 Not Allowed < Allow: OPTIONS, PUT < Content-Length: 0 < Date: Wed, 18 Nov 2020 22:41:46 GMT < Server: EC2ws < Connection: close < Content-Type: text/plain < * Closing connection 0 However, when executed from within a LXD container running inside the very same AWS/EC2 instance - it times out! root@lxd-container:~# curl -v http://169.254.169.254/latest/api/token * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) GET /latest/api/token HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/7.58.0 Accept: */* Even more weirdly, these work inside the container: curl -v http://169.254.169.254/latest/api/ curl -v http://169.254.169.254/latest/api/t curl -v http://169.254.169.254/latest/api/to curl -v http://169.254.169.254/latest/api/tok curl -v http://169.254.169.254/latest/api/toke And this times out: curl -v http://169.254.169.254/latest/api/token Does anyone know why? tcpdump doesn't give me many clues (TTL?). A somewhat related post (with docker having a similar issue): https://rtfm.co.ua/en/aws-eksctl-put-http-169-254-169-254-latest-api-token-net-http-request-canceled-2/ But, I'm no closer in getting a similar workaround for LXD. If someone's struggling with a similar issue - here is a fix: aws ec2 modify-instance-metadata-options --instance-id i-abcdefghijklmn --http-put-response-hop-limit 2 --http-endpoint enabled Tomasz Chmielewski https://lxadm.com ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] AWS EC2: timeout connecting to instance metadata webserver (169.254.169.254) for *some* URLs (when connecting from a LXD container)
On 2020-11-18 23:50, Tomasz Chmielewski wrote: That's a weird one! In AWS, there is a concept of "instance metadata" - a webserver which lets you fetch some instance metadata using http: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html For example, you can run this (from both AWS/EC2 instance and LXD container running inside a AWS/EC2 instance), it will return some metadata: curl -v http://169.254.169.254/latest/meta-data/ Now, some of these requests time out when executed from a LXD container running inside a AWS/EC2 - but work perfectly from the very same AWS/EC2 instance. For example, this request works fine from AWS/EC2 instance (ignore the output - HTTP connection works just fine): root@aws-instance:~# curl -v http://169.254.169.254/latest/api/token * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) GET /latest/api/token HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/7.58.0 Accept: */* < HTTP/1.1 405 Not Allowed < Allow: OPTIONS, PUT < Content-Length: 0 < Date: Wed, 18 Nov 2020 22:41:46 GMT < Server: EC2ws < Connection: close < Content-Type: text/plain < * Closing connection 0 However, when executed from within a LXD container running inside the very same AWS/EC2 instance - it times out! root@lxd-container:~# curl -v http://169.254.169.254/latest/api/token * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) GET /latest/api/token HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/7.58.0 Accept: */* Even more weirdly, these work inside the container: curl -v http://169.254.169.254/latest/api/ curl -v http://169.254.169.254/latest/api/t curl -v http://169.254.169.254/latest/api/to curl -v http://169.254.169.254/latest/api/tok curl -v http://169.254.169.254/latest/api/toke And this times out: curl -v http://169.254.169.254/latest/api/token Does anyone know why? tcpdump doesn't give me many clues (TTL?). A somewhat related post (with docker having a similar issue): https://rtfm.co.ua/en/aws-eksctl-put-http-169-254-169-254-latest-api-token-net-http-request-canceled-2/ But, I'm no closer in getting a similar workaround for LXD. Tomasz Chmielewski ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
