Thanks Donald. In summary: spark-streaming application (ZTT) is used to read message from Kafka, unpacks the data and write the data in OpenTSDB. In case of HDFS store, Gobblin can be used with no additional program.
In Gobblin case, what is the role of PNDA? Is it that PNDA developed a plugin in Gobblin? If so, what does that do? Thanks Srini From: donaldh via lists.onap.org [mailto:[email protected]] Sent: Monday, November 26, 2018 2:45 AM To: Addepalli, Srinivasa R <[email protected]>; [email protected] Subject: Re: [onap-discuss] PNDA Demo - Few Questions Answered #dcaegen2 Yes, gobblin is configured to automatically ingest data that it recognises off any topic into HDFS. PNDA can normalize incoming data to Avro but gobblin would need to be configured to do so. PNDA currently runs gobblin in batch mode, scheduled every 30 minutes by default. Each batch produces a separate file in HDFS. PNDA can be used to ingest data from Kafka to OpenTSDB. Gobblin is not used for this, though that may be possible. The pnda-ztt-app in the dcaegen2/analytics/pnda repository is a spark streaming application that receives data from Kafka, unpacks timeseries data values and writes them to OpenTSDB. The unpacking step is schema specific so the app is extensible. It currently unpacks Cisco telemetry and VES cpu messages. It will be extended to support all VES message content. Cheers, Donald. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14045): https://lists.onap.org/g/onap-discuss/message/14045 Mute This Topic: https://lists.onap.org/mt/28240683/21656 Mute #dcaegen2: https://lists.onap.org/mk?hashtag=dcaegen2&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
