Hi Srini, PNDA uses gobblin to ingest data from kafka topics into the HDFS datastore. Prior to version 5.0, PNDA would only recognise data encapsulated in the PNDA AVRO schema so the data had to be encoded externally – either at source, or using an intermediary like Logstash. The AVRO fields provide dataset metadata, to provide the dataset name, where and when received. So, as you mentioned, logstash was being used to relay messages from DMaaP to PNDA-kafka _and_ encapsulate the data in AVRO – i.e. Kafka –> AVRO.encode –> Kafka.
With PNDA 5.0 it is possible to configure gobblin to recognise kafka messages that are not AVRO encoded. For example, it can understand protobuf encoded messages so is able to directly ingest protobuf encoded telemetry messages. It can also directly ingest JSON, with some additional configuration to allow gobblin to identify the metadata, i.e. dataset name, source and timestamp. Gobblin configuration is described here: http://pnda.io/pnda-guide/streamingest/topic-preparation.html#gobblin-topic-configuration We could configure gobblin to directly recognise VES events either by topic name, or by reading the VES messages. This has the effect of moving the AVRO encoding step to be internal to the PNDA platform, but would still require a bridge from DMaaP-kafka to PNDA-kafka. We should be able to use the Kafka Mirror Maker for this but should evaluate this solution in a Kubernetes deployment environment. Topics we need to cover in more detail are: * Which data from DMaaP should be ingested into PNDA HDFS – working assumption is the VES topic. * What is the best technical solution for transferring data from DMaaP to PNDA – assumption is Kafka Mirror Maker. * What assumptions can we make about encoding of data that gets delivered to PNDA – answer depends on first bullet point. Cheers, Donald. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13847): https://lists.onap.org/g/onap-discuss/message/13847 Mute This Topic: https://lists.onap.org/mt/28194418/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
