Thank you Donald. Understood. But, I need to dig deeper myself in Gobblin and Kafka connector to understanding their capabilities.
You mentioned about HDFS and how Gobblin can be used to ingest the data from Kafka topic to HDFS. As I understand from your explanation, PNDA with Gobblin can even normalize the incoming data encoding to Avro before storing in HDFS. In case of HDFS, does it work in the same fashion as spark-streaming in terms of micro batching and writing each micro batch as a separate file in HDFS? Another question: Can PNDA+Gobblin be used to ingest the data from Kafka in OpenTSDB too? Thanks Srini From: [email protected] [mailto:[email protected]] On Behalf Of Donald Hunter via Lists.Onap.Org Sent: Monday, November 19, 2018 3:56 AM To: [email protected] Subject: [onap-discuss] PNDA Demo - Few Questions Answered #dcaegen2 Reposting to add #dcaegen2 hashtag Hi Cristina and Frank, In the demo meeting, I heard you mentioning Gobblin project. In the demo, you are using logstash to - Kafka as input - Transform the data to Avro format - Using Kafka as output Is the intention of mentioning Gobblin project is to indicate the replacement of logstash with Gobblin? Thanks Srini Hi Srini, PNDA uses gobblin to ingest data from kafka topics into the HDFS datastore. Prior to version 5.0, PNDA would only recognise data encapsulated in the PNDA AVRO schema so the data had to be encoded externally – either at source, or using an intermediary like Logstash. The AVRO fields provide dataset metadata, to provide the dataset name, where and when received. So, as you mentioned, logstash was being used to relay messages from DMaaP to PNDA-kafka _and_ encapsulate the data in AVRO – i.e. Kafka –> AVRO.encode –> Kafka. With PNDA 5.0 it is possible to configure gobblin to recognise kafka messages that are not AVRO encoded. For example, it can understand protobuf encoded messages so is able to directly ingest protobuf encoded telemetry messages. It can also directly ingest JSON, with some additional configuration to allow gobblin to identify the metadata, i.e. dataset name, source and timestamp. Gobblin configuration is described here: http://pnda.io/pnda-guide/streamingest/topic-preparation.html#gobblin-topic-configuration We could configure gobblin to directly recognise VES events either by topic name, or by reading the VES messages. This has the effect of moving the AVRO encoding step to be internal to the PNDA platform, but would still require a bridge from DMaaP-kafka to PNDA-kafka. We should be able to use the Kafka Mirror Maker for this but should evaluate this solution in a Kubernetes deployment environment. Topics we need to cover in more detail are: * Which data from DMaaP should be ingested into PNDA HDFS – working assumption is the VES topic. * What is the best technical solution for transferring data from DMaaP to PNDA – assumption is Kafka Mirror Maker. * What assumptions can we make about encoding of data that gets delivered to PNDA – answer depends on first bullet point. Cheers, Donald. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14017): https://lists.onap.org/g/onap-discuss/message/14017 Mute This Topic: https://lists.onap.org/mt/28240683/21656 Mute #dcaegen2: https://lists.onap.org/mk?hashtag=dcaegen2&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
