Dear Logging team,
I am investigating the issue of the missing multicloud logging from kinaba
dashboard. I tried to generate log output from multicloud services, then dump
the output from filebeat sidecar for that pod, it seems something wrong when
the filebeat is trying to publish the logs to logstash services?
Could you help me identify the root cause of this issue? Thanks
root@oom-rancher:~# kubectl -n onap logs
dev-multicloud-multicloud-windriver-756db78d6b-x2895 filebeat-onap
2018/11/15 05:41:03.977066 log.go:91: INFO Harvester started for file:
/var/log/onap/multicloud/openstack/windriver/titanium_cloud.log
2018/11/15 05:41:27.621872 metrics.go:39: INFO Non-zero metrics in the last
30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1
filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1
libbeat.logstash.publish.read_bytes=12
libbeat.logstash.publish.write_bytes=2661
libbeat.logstash.published_and_acked_events=15
libbeat.publisher.published_events=15 publish.events=16
registrar.states.update=16 registrar.writes=1
2018/11/15 05:41:57.624578 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:42:27.621832 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:42:57.621846 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:43:11.449505 sync.go:85: ERR Failed to publish events caused by:
write tcp 10.42.166.152:36792->10.43.172.237:5044: write: connection reset by
peer
2018/11/15 05:43:11.449536 single.go:91: INFO Error publishing events
(retrying): write tcp 10.42.166.152:36792->10.43.172.237:5044: write:
connection reset by peer
2018/11/15 05:43:27.621866 metrics.go:39: INFO Non-zero metrics in the last
30s: libbeat.logstash.call_count.PublishEvents=2
libbeat.logstash.publish.read_bytes=12
libbeat.logstash.publish.write_bytes=2605
libbeat.logstash.publish.write_errors=1
libbeat.logstash.published_and_acked_events=15
libbeat.logstash.published_but_not_acked_events=15
libbeat.publisher.published_events=15 publish.events=15
registrar.states.update=15 registrar.writes=1
2018/11/15 05:43:57.621849 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:44:27.621863 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:44:57.621870 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:45:27.621862 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:45:57.621852 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:46:27.622513 metrics.go:34: INFO No non-zero metrics in the last
30s
2018/11/15 05:46:56.461847 sync.go:85: ERR Failed to publish events caused by:
write tcp 10.42.166.152:39602->10.43.172.237:5044: write: connection reset by
peer
2018/11/15 05:46:56.463016 single.go:91: INFO Error publishing events
(retrying): write tcp 10.42.166.152:39602->10.43.172.237:5044: write:
connection reset by peer
2018/11/15 05:46:57.621889 metrics.go:39: INFO Non-zero metrics in the last
30s: libbeat.logstash.call_count.PublishEvents=2
libbeat.logstash.publish.read_bytes=18
libbeat.logstash.publish.write_bytes=3750
libbeat.logstash.publish.write_errors=1
libbeat.logstash.published_and_acked_events=15
libbeat.logstash.published_but_not_acked_events=15
libbeat.publisher.published_events=15 publish.events=15
registrar.states.update=15 registrar.writes=1
root@oom-rancher:~# kubectl describe
pod/dev-multicloud-multicloud-windriver-756db78d6b-x2895 -n onap
Name: dev-multicloud-multicloud-windriver-756db78d6b-x2895
Namespace: onap
Node: oom-k8s-07/
Start Time: Tue, 13 Nov 2018 17:18:06 +0000
Labels: app=multicloud-windriver
pod-template-hash=3128634826
release=dev-multicloud
Annotations: sidecar.istio.io/inject=true
Status: Running
IP: 10.42.166.152
Controlled By: ReplicaSet/dev-multicloud-multicloud-windriver-756db78d6b
Containers:
multicloud-windriver:
Container ID:
docker://3bdff3f932e0a41dcc902b529fe7f10b4ae6c3ce8b1a2f13b28e08b40a5be917
Image:
10.12.5.2:5000/onap/multicloud/openstack-windriver:1.2.1-STAGING
Image ID:
docker-pullable://10.12.5.2:5000/onap/multicloud/openstack-windriver@sha256:52c6ec425bdaf3066e03e41cceffe15dffc7b6c0833e19c518ccb0aea0a26d44
Port: 9005/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 13 Nov 2018 17:18:27 +0000
Ready: True
Restart Count: 0
Liveness: http-get
http://:9005/api/multicloud-titaniumcloud/v1/swagger.json delay=30s timeout=10s
period=10s #success=1 #failure=5
Environment:
MSB_ADDR: msb-iag.onap
MSB_PORT: 80
AAI_ADDR: aai.onap
AAI_PORT: 8443
AAI_SCHEMA_VERSION: v13
AAI_USERNAME: AAI
AAI_PASSWORD: AAI
Mounts:
/opt/windriver/titanium_cloud/pub/config/log.yml from windriver-logconfig
(rw)
/var/log/onap from windriver-log (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xv4db
(ro)
filebeat-onap:
Container ID:
docker://718d9c339f11da73279fb6e115ad722fca546c5addfae3533d2bddb8f6b31e43
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID:
docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 13 Nov 2018 17:18:27 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/filebeat/data from windriver-data-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from windriver-log (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xv4db
(ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
windriver-log:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
windriver-data-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: multicloud-filebeat-configmap
Optional: false
windriver-logconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-multicloud-multicloud-windriver-log-configmap
Optional: false
default-token-xv4db:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xv4db
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Best Regards,
Bin Yang, Solution Engineering Team, Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189
Skype: yangbincs993
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13751): https://lists.onap.org/g/onap-discuss/message/13751
Mute This Topic: https://lists.onap.org/mt/28143443/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
