Hi Michael, Appreciate your quick response.
Yes, I just created a Jira issue to track this issue: https://jira.onap.org/browse/LOG-842 ### make sure the multicloud service is up and running: root@oom-rancher:~# kubectl -n onap get pods |grep multicloud-windriver The command line to generate multicloud log looks like: ######## OOM based ONAP export OPENO_IP=<k8s node IP> export MC_EP_PORT=30294 export MC_EP=http://$OPENO_IP:$MC_EP_PORT/api/multicloud-titaniumcloud/v1/CloudOwner2/RegionOne curl -v -s -H "Content-Type: application/json" -X POST -d '{}' $MC_EP/identity/v2.0/tokens root@oom-rancher:~# kubectl -n onap logs dev-multicloud-multicloud-windriver-756db78d6b-x2895 filebeat-onap Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: Michael O'Brien [mailto:frank.obr...@amdocs.com] Sent: Thursday, November 15, 2018 8:43 PM To: Yang, Bin; onap-discuss@lists.onap.org Cc: Ethan Lynn (ethanly...@vmware.com); HUANG, HAIBIN (haibin.hu...@intel.com); Sudhakar Reddy; Morales, Victor Subject: RE: [onap-discuss][logging][multicloud] filebeat sidecar for multicloud pods failed to publish log message to logstash Bin, Hi, yes until the log-ls service is in liveness state and ready to receive logs through the beats plugin – logs won’t appear in the elk stack For all the 3 kibana pods in 3 of the onap components – kibana takes a while to come up – usually 2 min after the corresponding es pod. Do you have a jira tracking all this that we can attach work to. You can query the elasticsearch pod on the command line or run a gui query in the kibana ui after the index is saved. Instructions on https://wiki.onap.org/display/DW/Logging+Developer+Guide Do you have the rest call or action that generated your logs so I can reproduce it in my stack. Thank you /michael From: Yang, Bin <bin.y...@windriver.com> Sent: Thursday, November 15, 2018 1:22 AM To: onap-discuss@lists.onap.org; Yang, Bin <bin.y...@windriver.com>; Michael O'Brien <frank.obr...@amdocs.com> Cc: Ethan Lynn (ethanly...@vmware.com) <ethanly...@vmware.com>; HUANG, HAIBIN (haibin.hu...@intel.com) <haibin.hu...@intel.com>; Sudhakar Reddy <sudhakar.re...@amdocs.com>; Morales, Victor <victor.mora...@intel.com> Subject: RE: [onap-discuss][logging][multicloud] filebeat sidecar for multicloud pods failed to publish log message to logstash Hi, I think I found the root cause: The logstash service is not up yet. So I changed to another oom instance for checking: Integration-SB-06. This time there is no such error reported. According to the log of filebeat below, I believe the log message was published to logstash service, but still I cannot discover the multicloud log message from kibana: http://10.12.6.68:30253/app/kibana Could you shed some light on this issue? Thanks 2018/11/15 06:11:11.680805 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:11:41.677506 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:12:11.675564 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:12:31.206613 log.go:91: INFO Harvester started for file: /var/log/onap/multicloud/openstack/windriver/titanium_cloud.log 2018/11/15 06:12:41.679615 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=12 libbeat.logstash.publish.write_bytes=2618 libbeat.logstash.published_and_acked_events=15 libbeat.publisher.published_events=15 publish.events=16 registrar.states.update=16 registrar.writes=1 2018/11/15 06:13:11.724319 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:13:41.672813 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:14:11.674501 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:14:41.671870 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:15:11.688733 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:15:41.672633 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:16:11.674214 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:16:41.678199 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:17:11.673012 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:17:36.506016 log.go:116: INFO File is inactive: /var/log/onap/multicloud/openstack/windriver/titanium_cloud.log. Closing because close_inactive of 5m0s reached. 2018/11/15 06:17:41.672231 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.closed=1 filebeat.harvester.open_files=-1 filebeat.harvester.running=-1 publish.events=1 registrar.states.update=1 registrar.writes=1 2018/11/15 06:18:11.678847 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:18:41.676391 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:19:11.675123 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:19:41.671444 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 06:20:11.671873 metrics.go:34: INFO No non-zero metrics in the last 30s Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 From: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> [mailto:onap-discuss@lists.onap.org] On Behalf Of Yang Bin Sent: Thursday, November 15, 2018 1:57 PM To: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>; frank.obr...@amdocs.com<mailto:frank.obr...@amdocs.com> Cc: Ethan Lynn (ethanly...@vmware.com<mailto:ethanly...@vmware.com>); HUANG, HAIBIN (haibin.hu...@intel.com<mailto:haibin.hu...@intel.com>); Sudhakar Reddy (sudhakar.re...@amdocs.com<mailto:sudhakar.re...@amdocs.com>); Morales, Victor Subject: [onap-discuss][logging][multicloud] filebeat sidecar for multicloud pods failed to publish log message to logstash Dear Logging team, I am investigating the issue of the missing multicloud logging from kinaba dashboard. I tried to generate log output from multicloud services, then dump the output from filebeat sidecar for that pod, it seems something wrong when the filebeat is trying to publish the logs to logstash services? Could you help me identify the root cause of this issue? Thanks root@oom-rancher:~# kubectl -n onap logs dev-multicloud-multicloud-windriver-756db78d6b-x2895 filebeat-onap 2018/11/15 05:41:03.977066 log.go:91: INFO Harvester started for file: /var/log/onap/multicloud/openstack/windriver/titanium_cloud.log 2018/11/15 05:41:27.621872 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1 filebeat.harvester.running=1 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=12 libbeat.logstash.publish.write_bytes=2661 libbeat.logstash.published_and_acked_events=15 libbeat.publisher.published_events=15 publish.events=16 registrar.states.update=16 registrar.writes=1 2018/11/15 05:41:57.624578 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:42:27.621832 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:42:57.621846 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:43:11.449505 sync.go:85: ERR Failed to publish events caused by: write tcp 10.42.166.152:36792->10.43.172.237:5044: write: connection reset by peer 2018/11/15 05:43:11.449536 single.go:91: INFO Error publishing events (retrying): write tcp 10.42.166.152:36792->10.43.172.237:5044: write: connection reset by peer 2018/11/15 05:43:27.621866 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=12 libbeat.logstash.publish.write_bytes=2605 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=15 libbeat.logstash.published_but_not_acked_events=15 libbeat.publisher.published_events=15 publish.events=15 registrar.states.update=15 registrar.writes=1 2018/11/15 05:43:57.621849 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:44:27.621863 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:44:57.621870 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:45:27.621862 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:45:57.621852 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:46:27.622513 metrics.go:34: INFO No non-zero metrics in the last 30s 2018/11/15 05:46:56.461847 sync.go:85: ERR Failed to publish events caused by: write tcp 10.42.166.152:39602->10.43.172.237:5044: write: connection reset by peer 2018/11/15 05:46:56.463016 single.go:91: INFO Error publishing events (retrying): write tcp 10.42.166.152:39602->10.43.172.237:5044: write: connection reset by peer 2018/11/15 05:46:57.621889 metrics.go:39: INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=18 libbeat.logstash.publish.write_bytes=3750 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_and_acked_events=15 libbeat.logstash.published_but_not_acked_events=15 libbeat.publisher.published_events=15 publish.events=15 registrar.states.update=15 registrar.writes=1 root@oom-rancher:~# kubectl describe pod/dev-multicloud-multicloud-windriver-756db78d6b-x2895 -n onap Name: dev-multicloud-multicloud-windriver-756db78d6b-x2895 Namespace: onap Node: oom-k8s-07/ Start Time: Tue, 13 Nov 2018 17:18:06 +0000 Labels: app=multicloud-windriver pod-template-hash=3128634826 release=dev-multicloud Annotations: sidecar.istio.io/inject=true Status: Running IP: 10.42.166.152 Controlled By: ReplicaSet/dev-multicloud-multicloud-windriver-756db78d6b Containers: multicloud-windriver: Container ID: docker://3bdff3f932e0a41dcc902b529fe7f10b4ae6c3ce8b1a2f13b28e08b40a5be917 Image: 10.12.5.2:5000/onap/multicloud/openstack-windriver:1.2.1-STAGING Image ID: docker-pullable://10.12.5.2:5000/onap/multicloud/openstack-windriver@sha256:52c6ec425bdaf3066e03e41cceffe15dffc7b6c0833e19c518ccb0aea0a26d44 Port: 9005/TCP Host Port: 0/TCP State: Running Started: Tue, 13 Nov 2018 17:18:27 +0000 Ready: True Restart Count: 0 Liveness: http-get http://:9005/api/multicloud-titaniumcloud/v1/swagger.json delay=30s timeout=10s period=10s #success=1 #failure=5 Environment: MSB_ADDR: msb-iag.onap MSB_PORT: 80 AAI_ADDR: aai.onap AAI_PORT: 8443 AAI_SCHEMA_VERSION: v13 AAI_USERNAME: AAI AAI_PASSWORD: AAI Mounts: /opt/windriver/titanium_cloud/pub/config/log.yml from windriver-logconfig (rw) /var/log/onap from windriver-log (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv4db (ro) filebeat-onap: Container ID: docker://718d9c339f11da73279fb6e115ad722fca546c5addfae3533d2bddb8f6b31e43 Image: docker.elastic.co/beats/filebeat:5.5.0 Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 Port: <none> Host Port: <none> State: Running Started: Tue, 13 Nov 2018 17:18:27 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/filebeat/data from windriver-data-filebeat (rw) /usr/share/filebeat/filebeat.yml from filebeat-conf (rw) /var/log/onap from windriver-log (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv4db (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: windriver-log: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: windriver-data-filebeat: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: filebeat-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: multicloud-filebeat-configmap Optional: false windriver-logconfig: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-multicloud-multicloud-windriver-log-configmap Optional: false default-token-xv4db: Type: Secret (a volume populated by a Secret) SecretName: default-token-xv4db Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: <none> Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 “Amdocs’ email platform is based on a third-party, worldwide, cloud-based system. Any emails sent to Amdocs will be processed and stored using such system and are accessible by third party providers of such system on a limited basis. Your sending of emails to Amdocs evidences your consent to the use of such system and such processing, storing and access”. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13780): https://lists.onap.org/g/onap-discuss/message/13780 Mute This Topic: https://lists.onap.org/mt/28143443/21656 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
