Thank you Tim,
After I changed the kubernetes services that expose 8161 from ClsusterIP to
NodePort, i setup port forward to the exposed port (8161) but i am not able
to reach the console, here is the config and the cmd i used:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2021-08-23T15:32:37Z"
labels:
ActiveMQArtemis: ex-aao
application: ex-aao-app
name: ex-aao-wconsj-1-svc
namespace: myproject
ownerReferences:
- apiVersion: broker.amq.io/v2alpha5
blockOwnerDeletion: true
controller: true
kind: ActiveMQArtemis
name: ex-aao
uid: d35e1dd0-37ed-4b8e-8db4-cd7081d3502f
resourceVersion: "323731"
uid: f4866b76-6af8-4196-9d68-f90d535eb3dc
spec:
clusterIP: 10.107.150.161
clusterIPs:
- 10.107.150.161
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: wconsj-1
nodePort: 32061
port: 8161
protocol: TCP
targetPort: 8161
publishNotReadyAddresses: true
selector:
ActiveMQArtemis: ex-aao
application: ex-aao-app
statefulset.kubernetes.io/pod-name: ex-aao-ss-1
sessionAffinity: None
type: NodePort
status:
loadBalancer:
ingress:
- hostname: localhost
Forward cmd:
kubectl port-forward pod/ex-aao-ss-0 8161:8161 -n myproject
Per your suggestion, I attempted to bind the web service to 0.0.0.0 but i
have not yet found where the code swap out he default values
in
/tmp/remote_source/app/src/yacfg/profiles/artemis/2.18.0/_modules/bootstrap_xml/*
of the init image quay.io/artemiscloud/activemq-artemis-broker-init:0.2.6.
I gonna look into ingress next
Thai Le
On Mon, Aug 23, 2021 at 12:59 AM Tim Bain <tb...@alumni.duke.edu> wrote:
> Thanks for determining that port-forwarding directly to the pods doesn't
> work either. Would you mind checking whether port-forwarding to a NodePort
> service succeeds? I'm glad to hear that a NodePort service works as
> expected when accessed directly, but my goal in asking you to test that
> configuration was to try to see whether the problem is with the kubectl
> port-forward process, so doing that final step of the test would help
> assess that.
>
> I've got two possible theories for what's going on. One is that kubectl
> port-forward doesn't work correctly (or that you're doing something wrong
> when invoking it, though everything you've shown so far looks fine), in
> which case Artemis is fine but your attempt to use kubectl port-forward is
> where this is breaking. If that's the case, you'll want to take that up
> with the Kubernetes community (e.g. file a bug in the Kubernetes GitHub
> project <https://github.com/kubernetes/kubernetes>). The other possibility
> is that kubectl port-forward is working fine, but the Artemis process is
> binding to the TCP port in a way that doesn't result in it accepting
> incoming connections from whatever network path gets used for kubectl
> port-forward even though it's fine when accessed via the Kubernetes
> service. In that case, you'll want to tweak the bootstrap.xml to make it
> listen on 0.0.0.0 (i.e. all IP addresses associated with the pod) rather
> than the specific address you listed in your last message. I suspect the
> latter, but both are possible.
>
> If modifying the bootstrap.xml to make it listen on 0.0.0.0 is easy to do,
> please do that and see where it gets you. If that's non-trivial, then
> please deploy an Nginx deployment with a ClusterIP service in front of it (
>
> https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
> is one of many resources online that describe how to do this), confirm that
> you can hit the Nginx service from within the cluster, and then test
> whether you can use kubectl port-forward to the Nginx ClusterIP service.
> (Ideally, use local port 8161 when doing the kubectl port-forward, just to
> keep things as equivalent as possible.) If you can get through to the Nginx
> service via kubectl port-forward but can't do the same thing to the Artemis
> web console, then that to me would be a pretty clear indication that the
> issue is within the Artemis config and not a bug in kubectl, and at that
> point you'd want to pursue how to get Artemis to bind to 0.0.0.0 even if
> that's non-trivial to do. Once you do that, I'd expect you to be able to
> port-forward to either the service or to each individual pods as needed.
>
> As for why it didn't work when you added a host entry to map to 127.0.0.1,
> that's because that hostname isn't your Windows machine, it's the container
> that's running on your Windows machine, which has its own network stack and
> address. Networking is the most complicated part of working with
> Kubernetes, and it's easy to get confused about exactly which host is
> "localhost" in a given context, but if you think of each container as being
> on a different machine rather than the physical machine you're actually
> running on, that will help avoid at least some of the places where
> confusion could occur.
>
> One other thing: I'd encourage you to think about whether kubectl
> port-forward is really the right tool for the job here. It's a great way to
> do ad hoc troubleshooting, but if you're going to be using it frequently
> and/or if you might want to use it for automated monitoring in production,
> I still think you might be better off with an always-available ingress
> route(s) that lets you expose these things on a permanent basis (whether
> that's exposing the service that load-balances across all brokers or
> whether you set up individual services for each individual pod and then
> expose each of those single-broker services so you can access the web
> console of any broker). And yes, ingress is a feature by which incoming
> HTTP traffic is routed to resources (certainly services, but I'm not sure
> what other resource types can be exposed) within Kubernetes.
>
> Tim
>
>
> On Fri, Aug 20, 2021 at 9:43 AM Thai Le <lnthai2...@gmail.com> wrote:
>
> > I tried port-forward directly to the pod, it didn't work:
> > C:\Users\nle>kubectl port-forward pod/ex-aao-ss-0 8161:8161 -n myproject
> > Forwarding from 127.0.0.1:8161 -> 8161
> > Forwarding from [::1]:8161 -> 8161
> > Handling connection for 8161
> > Handling connection for 8161
> > E0820 09:06:52.508157 13064 portforward.go:400] an error occurred
> > forwarding 8161 -> 8161: error forwarding port 8161 to pod
> > ff6c35be7d3ace906d726b6abd38dbe122ddf2530647a44c799ca9a8a15ab245, uid :
> > exit status 1: 2021/08/20 13:06:46 socat[31487] E connect(17, AF=2
> > 127.0.0.1:8161, 16): Connection refused
> > E0820 09:06:52.522192 13064 portforward.go:400] an error occurred
> > forwarding 8161 -> 8161: error forwarding port 8161 to pod
> > ff6c35be7d3ace906d726b6abd38dbe122ddf2530647a44c799ca9a8a15ab245, uid :
> > exit status 1: 2021/08/20 13:06:46 socat[31488] E connect(17, AF=2
> > 127.0.0.1:8161, 16): Connection refused
> >
> > Since the bootstrap.xml indicates the web server is binding to
> > http://ex-aao-ss-0.ex-aao-hdls-svc.myproject.svc.cluster.local:8161, I
> > also
> > tried to add the
> "ex-aao-ss-0.ex-aao-hdls-svc.myproject.svc.cluster.local"
> > to my windows hosts file which map it to 127.0.0.1 then try access it
> from
> > my desktop browser but it's still not working.
> >
> > If i change the 2 ClusterIP services (*ex-aao-wconsj-0-svc* and
> > *ex-aao-wconsj-1-svc*) that expose 8161 for the 2 brokers to NodePort
> then
> > I can access the web console directly from outside the cluster without
> port
> > forwarding. I don't understand why this works but direct port forwarding
> to
> > the pod does not.
> >
> > The purpose of this exercise is to make sure that when developing the
> > application locally we can point to artemis running on a cluster and
> > observe/debug message distribution between multiple artemis brokers. In
> > production, we also need access to the console of each broker at the same
> > time for troubleshooting, my original thought is just port-forward to
> each
> > pod when needed. Forgive me for my limited knowledge of kubernetes, but
> as
> > I understand, ingress is to load balance http traffic so at one point in
> > time, the console of a particular broker can be accessed.
> >
> > Thai Le
> >
> >
> > On Fri, Aug 20, 2021 at 12:14 AM Tim Bain <tb...@alumni.duke.edu> wrote:
> >
> > > Can you port-forward directly to the individual pods successfully? If
> > that
> > > doesn't work, then going through the service won't, so make sure that
> > > building block is working.
> > >
> > > Also, if you switch the service to be a NodePort service, can you hit
> the
> > > web console from outside the K8s cluster without the port-forward? And
> > > assuming that works, can you port-forward against that service
> > > successfully? I'm not proposing you make that a permanent change, just
> > > suggesting you try these variations to attempt to characterize the
> > problem.
> > >
> > > One other question: long-term, do you plan to expose the web console
> port
> > > outside of the cluster? If so, you won't (shouldn't) be using kubectl
> > > port-forward for that, and you should probably be using an ingress
> proxy,
> > > so maybe just set that up and don't worry about getting the
> port-forward
> > > approach to work.
> > >
> > > Tim
> > >
> > > On Wed, Aug 18, 2021, 1:24 PM Thai Le <lnthai2...@gmail.com> wrote:
> > >
> > > > Thank you Justin for your suggestion.
> > > >
> > > > I looked at the bootstrap.xml of both broker nodes and the binding is
> > set
> > > > to the hostname of the pod:
> > > > <web bind="
> > > > http://ex-aao-ss-0.ex-aao-hdls-svc.myproject.svc.cluster.local:8161";
> > > > path="web">
> > > > <web bind="
> > > > http://ex-aao-ss-1.ex-aao-hdls-svc.myproject.svc.cluster.local:8161";
> > > > path="web">
> > > > So it makes sense that I got a connection refused when accessing the
> > pod
> > > > from my desktop using localhost through port forwarding to the pod.
> > > >
> > > > I also see that there are 3 kubernetes services running, one for both
> > > 8161
> > > > and 61616 (I think this is the main service that i can hit from the
> jms
> > > > consumer) and 2 other that only for 8161 but for each broker node (I
> > > > believe this is to allow clients from outside kubernetes to access
> the
> > > web
> > > > console using IP, giving that routing from outside cluster to the
> > service
> > > > IP is present):
> > > > kubectl get services -n myproject
> > > > NAME TYPE CLUSTER-IP EXTERNAL-IP
> > > > PORT(S) AGE
> > > > activemq-artemis-operator ClusterIP 10.100.240.205 <none>
> > > > 8383/TCP 46h
> > > > ex-aao-hdls-svc ClusterIP None <none>
> > > > 8161/TCP,61616/TCP 134m
> > > > ex-aao-ping-svc ClusterIP None <none>
> > > > 8888/TCP 134m
> > > > ex-aao-wconsj-0-svc ClusterIP *10.96.183.20* <none>
> > > > 8161/TCP 134m
> > > > ex-aao-wconsj-1-svc ClusterIP *10.98.233.91* <none>
> > > > 8161/TCP 134m
> > > >
> > > > Here is the description of the main service:
> > > > kubectl describe service ex-aao-hdls-svc -n myproject
> > > > Name: * ex-aao-hdls-svc*
> > > > Namespace: myproject
> > > > Labels: ActiveMQArtemis=ex-aao
> > > > application=ex-aao-app
> > > > Annotations: <none>
> > > > Selector: ActiveMQArtemis=ex-aao,application=ex-aao-app
> > > > Type: ClusterIP
> > > > IP Family Policy: SingleStack
> > > > IP Families: IPv4
> > > > IP: None
> > > > IPs: None
> > > > Port: console-jolokia 8161/TCP
> > > > TargetPort: 8161/TCP
> > > > Endpoints: *10.1.0.30*:8161,*10.1.0.31*:8161
> > > > Port: all 61616/TCP
> > > > TargetPort: 61616/TCP
> > > > Endpoints: *10.1.0.30*:61616,*10.1.0.31*:61616
> > > > Session Affinity: None
> > > > Events: <none>
> > > >
> > > > And here is the description the other 2 services:
> > > > kubectl describe service ex-aao-wconsj-0-svc -n myproject
> > > > Name: ex-aao-wconsj-0-svc
> > > > Namespace: myproject
> > > > Labels: ActiveMQArtemis=ex-aao
> > > > application=ex-aao-app
> > > > Annotations: <none>
> > > > Selector: ActiveMQArtemis=ex-aao,application=ex-aao-app,
> > > > statefulset.kubernetes.io/pod-name=ex-aao-ss-0
> > > > Type: ClusterIP
> > > > IP Family Policy: SingleStack
> > > > IP Families: IPv4
> > > > IP: *10.96.183.20*
> > > > IPs: *10.96.183.20*
> > > > Port: wconsj-0 8161/TCP
> > > > TargetPort: 8161/TCP
> > > > Endpoints: *10.1.0.30*:8161
> > > > Session Affinity: None
> > > > Events: <none>
> > > >
> > > > kubectl describe service ex-aao-wconsj-1-svc -n myproject
> > > > Name: ex-aao-wconsj-1-svc
> > > > Namespace: myproject
> > > > Labels: ActiveMQArtemis=ex-aao
> > > > application=ex-aao-app
> > > > Annotations: <none>
> > > > Selector: ActiveMQArtemis=ex-aao,application=ex-aao-app,
> > > > statefulset.kubernetes.io/pod-name=ex-aao-ss-1
> > > > Type: ClusterIP
> > > > IP Family Policy: SingleStack
> > > > IP Families: IPv4
> > > > IP: *10.98.233.91*
> > > > IPs: *10.98.233.91*
> > > > Port: wconsj-1 8161/TCP
> > > > TargetPort: 8161/TCP
> > > > Endpoints: *10.1.0.31*:8161
> > > > Session Affinity: None
> > > > Events: <none>
> > > >
> > > > The 2 pods hosting the broker nodes are ex-aao-ss0 and ex-aao-ss1:
> > > > kubectl get all -o wide -n myproject
> > > > NAME READY STATUS
> > > RESTARTS
> > > > AGE IP
> > > > pod/activemq-artemis-operator-bb9cf6567-qjdzs 1/1 Running 0
> > > > 46h 10.1.0.6
> > > > pod/debug 1/1 Running 0
> > > > 162m 10.1.0.29
> > > > pod/ex-aao-ss-0 1/1 Running 0
> > > > 155m *10.1.0.30*
> > > > pod/ex-aao-ss-1 1/1 Running 0
> > > > 154m *10.1.0.31*
> > > >
> > > > Hence, from another pod in the same cluster I can access the web
> > console
> > > :
> > > > curl -L http://*ex-aao-hdls-svc*:8161, so I should be able to port
> > > forward
> > > > using this service instead of the pod:
> > > > C:\Users\nle>kubectl port-forward service/ex-aao-hdls-svc 8161:8161
> -n
> > > > myproject
> > > > Forwarding from 127.0.0.1:8161 -> 8161
> > > > Forwarding from [::1]:8161 -> 8161
> > > >
> > > > However, hitting http://localhost:8161 from my desktop still give
> the
> > > same
> > > > error:
> > > >
> > > > Handling connection for 8161
> > > > Handling connection for 8161
> > > > E0818 14:51:30.135226 18024 portforward.go:400] an error occurred
> > > > forwarding 8161 -> 8161: error forwarding port 8161 to pod
> > > > ff6c35be7d3ace906d726b6abd38dbe122ddf2530647a44c799ca9a8a15ab245,
> uid :
> > > > exit status 1: 2021/08/18 18:51:26 socat[1906] E connect(17, AF=2
> > > > 127.0.0.1:8161, 16): Connection refused
> > > > E0818 14:51:30.136855 18024 portforward.go:400] an error occurred
> > > > forwarding 8161 -> 8161: error forwarding port 8161 to pod
> > > > ff6c35be7d3ace906d726b6abd38dbe122ddf2530647a44c799ca9a8a15ab245,
> uid :
> > > > exit status 1: 2021/08/18 18:51:26 socat[1907] E connect(17, AF=2
> > > > 127.0.0.1:8161, 16): Connection refused
> > > >
> > > > Do you have any other suggestion?
> > > >
> > > > Thai Le
> > > >
> > > >
> > > > On Wed, Aug 18, 2021 at 2:10 PM Justin Bertram <jbert...@apache.org>
> > > > wrote:
> > > >
> > > > > If the embedded web server which serves the console (as configured
> in
> > > > > bootstrap.xml) is bound to localhost then it will never be
> accessible
> > > > from
> > > > > a remote machine. You need to bind it to an IP or hostname which is
> > > > > externally accessible.
> > > > >
> > > > >
> > > > > Justin
> > > > >
> > > > > On Tue, Aug 17, 2021 at 2:58 PM Thai Le <lnthai2...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I am not sure if questions regarding Artemis cloud can be asked
> > here
> > > > but
> > > > > > since i found no mailing list for artemis cloud and the slack
> > channel
> > > > > needs
> > > > > > an invitation to join I'm gonna try my luck here.
> > > > > >
> > > > > > I installed the Artemis operator and an ActiveMQArtemis with a
> > > > deployment
> > > > > > plan of 2 brokers on my single node kubernetes (docker-desktop),
> > here
> > > > is
> > > > > > the deployment:
> > > > > >
> > > > > > apiVersion: broker.amq.io/v2alpha5
> > > > > > kind: ActiveMQArtemis
> > > > > > metadata:
> > > > > > name: ex-aao
> > > > > > spec:
> > > > > > adminUser: brokerAdmin
> > > > > > adminPassword: verySecret
> > > > > > deploymentPlan:
> > > > > > size: 2
> > > > > > image: placeholder
> > > > > > podSecurity:
> > > > > > runAsUser: 0
> > > > > > console:
> > > > > > expose: true
> > > > > > sslEnabled: false
> > > > > >
> > > > > > The 2 brokers are running and I can curl the web console from
> > another
> > > > pod
> > > > > > in the same kubernetes cluster. However, I cannot access the web
> > > > console
> > > > > > from my desktop (http://localhost:8161/console). I also tried to
> > > port
> > > > > > forward requests to port 8161 from my desktop to one of the 2
> > artemis
> > > > > pods
> > > > > > but it does not work either.
> > > > > >
> > > > > > I would appreciate it if anyone could give me a hint as to what
> may
> > > be
> > > > > > wrong or a direction to artemis cloud mailing list
> > > > > >
> > > > > > Thai Le
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Where there is will, there is a way
> > > >
> > >
> >
> >
> > --
> > Where there is will, there is a way
> >
>
--
Where there is will, there is a way
