BFergerson opened a new issue #6234: URL: https://github.com/apache/skywalking/issues/6234
- Why do you submit this issue? - [x] Question or discussion - [ ] Bug - [ ] Requirement - [ ] Feature or performance improvement ___ ### Question I tried to open this issue on https://github.com/apache/skywalking-kubernetes but there is no place for it so I'm asking here. I'm setting up skywalking-kubernetes from a bare virtual machine using these commands: ```sh 1 curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 2 chmod 700 get_helm.sh 3 ./get_helm.sh 4 sudo snap install microk8s --classic 5 sudo snap install microk8s --classic --channel=1.20/stable 6 microk8s.status --wait-ready 7 snap alias microk8s.kubectl kubectl 8 microk8s.kubectl config view --raw > $HOME/.kube/config 9 git clone https://github.com/apache/skywalking-kubernetes 10 cd skywalking-kubernetes/chart 11 helm repo add elastic https://helm.elastic.co 12 helm dep up skywalking 13 export SKYWALKING_RELEASE_NAME=skywalking 14 export SKYWALKING_RELEASE_NAMESPACE=default 15 helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" -f ./skywalking/values-es7.yaml 16 kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml 17 cat <<EOF | kubectl apply -f - apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard EOF 18 cat <<EOF | kubectl apply -f - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard EOF 19 kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}" 20 kubectl proxy --address=0.0.0.0 21 history ``` Once the above is complete, I view the status of the deployment via these urls: - http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login - http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/log/default/elasticsearch-master-0/pod?namespace=default&container=elasticsearch But what I end up seeing is a failed deployment of elasticsearch-master-0 with these logs: ``` OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. {"type": "server", "timestamp": "2021-01-19T22:27:19,936Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [73.5gb], net total_space [77.3gb], types [overlay]" } {"type": "server", "timestamp": "2021-01-19T22:27:19,949Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "heap size [1015.6mb], compressed ordinary object pointers [true]" } {"type": "server", "timestamp": "2021-01-19T22:27:19,952Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "node name [elasticsearch-master-0], node ID [OCY_MOKrSbG3f6qQKaGh0g], cluster name [elasticsearch]" } {"type": "server", "timestamp": "2021-01-19T22:27:19,954Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "version[7.5.1], pid[1], build[default/docker/3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96/2019-12-16T22:57:37.835892Z], OS[Linux/5.4.0-51-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]" } {"type": "server", "timestamp": "2021-01-19T22:27:19,957Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM home [/usr/share/elasticsearch/jdk]" } {"type": "server", "timestamp": "2021-01-19T22:27:19,958Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-959428078429237545, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32, filesize=64m, -Des.cgroups.hierarchy.override=/, -Xmx1g, -Xms1g, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,014Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [aggs-matrix-stats]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,014Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [analysis-common]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,014Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [flattened]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [frozen-indices]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-common]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-geoip]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [ingest-user-agent]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-expression]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-mustache]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,015Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [lang-painless]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [mapper-extras]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [parent-join]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [percolator]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [rank-eval]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [reindex]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [repository-url]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,016Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [search-business-rules]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,017Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [spatial]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,017Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transform]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,017Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [transport-netty4]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,017Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [vectors]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,017Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-analytics]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,017Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ccr]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,018Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-core]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,018Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-deprecation]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,018Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-enrich]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,018Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-graph]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,023Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ilm]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,023Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-logstash]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,023Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-ml]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,023Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-monitoring]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,023Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-rollup]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,024Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-security]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,024Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-sql]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,024Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-voting-only-node]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,024Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "loaded module [x-pack-watcher]" } {"type": "server", "timestamp": "2021-01-19T22:27:27,026Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "no plugins loaded" } {"type": "server", "timestamp": "2021-01-19T22:27:39,307Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" } {"type": "server", "timestamp": "2021-01-19T22:27:40,990Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "[controller/87] [Main.cc@110] controller (64 bit): Version 7.5.1 (Build ae3c3c51b849be) Copyright (c) 2019 Elasticsearch BV" } {"type": "server", "timestamp": "2021-01-19T22:27:42,679Z", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" } {"type": "server", "timestamp": "2021-01-19T22:27:43,086Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "using discovery type [zen] and seed hosts providers [settings]" } {"type": "server", "timestamp": "2021-01-19T22:27:45,838Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "initialized" } {"type": "server", "timestamp": "2021-01-19T22:27:45,839Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "starting ..." } {"type": "server", "timestamp": "2021-01-19T22:27:46,270Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "publish_address {10.1.137.4:9300}, bound_addresses {[::]:9300}" } {"type": "server", "timestamp": "2021-01-19T22:27:46,748Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" } {"type": "server", "timestamp": "2021-01-19T22:27:46,943Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to resolve host [elasticsearch-master-headless]", "stacktrace": ["java.net.UnknownHostException: elasticsearch-master-headless: Name or service not known", "at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]", "at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:930) ~[?:?]", "at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1499) ~[?:?]", "at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:849) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1489) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1348) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1282) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:528) ~[elasticsearch-7.5.1.jar:7.5.1]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:470) ~[elasticsearch-7.5.1.jar:7.5.1]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:803) ~[elasticsearch-7.5.1.jar:7.5.1]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.5.1.jar:7.5.1]", "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.1.jar:7.5.1]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } {"type": "server", "timestamp": "2021-01-19T22:27:47,822Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to resolve host [elasticsearch-master-headless]", "stacktrace": ["java.net.UnknownHostException: elasticsearch-master-headless", "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:798) ~[?:?]", "at java.net.InetAddress.getAllByName0(InetAddress.java:1489) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1348) ~[?:?]", "at java.net.InetAddress.getAllByName(InetAddress.java:1282) ~[?:?]", "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:528) ~[elasticsearch-7.5.1.jar:7.5.1]", "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:470) ~[elasticsearch-7.5.1.jar:7.5.1]", "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:803) ~[elasticsearch-7.5.1.jar:7.5.1]", "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.5.1.jar:7.5.1]", "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.1.jar:7.5.1]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:830) [?:?]"] } ``` - What do you want to know? What is causing `java.net.UnknownHostException: elasticsearch-master-headless` and how should I fix it? Also, what is the difference between https://github.com/apache/skywalking-kubernetes and https://github.com/apache/skywalking-swck. Is skywalking-swck an extension of skywalking-kubernetes which provides a CLI for easier control? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
