Yes, gobblin is configured to automatically ingest data that it recognises off any topic into HDFS. PNDA can normalize incoming data to Avro but gobblin would need to be configured to do so. PNDA currently runs gobblin in batch mode, scheduled every 30 minutes by default. Each batch produces a separate file in HDFS.
PNDA can be used to ingest data from Kafka to OpenTSDB. Gobblin is not used for this, though that may be possible. The pnda-ztt-app in the dcaegen2/analytics/pnda repository is a spark streaming application that receives data from Kafka, unpacks timeseries data values and writes them to OpenTSDB. The unpacking step is schema specific so the app is extensible. It currently unpacks Cisco telemetry and VES cpu messages. It will be extended to support all VES message content. Cheers, Donald. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14037): https://lists.onap.org/g/onap-discuss/message/14037 Mute This Topic: https://lists.onap.org/mt/28240683/21656 Mute #dcaegen2: https://lists.onap.org/mk?hashtag=dcaegen2&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
