Re: [onap-discuss] [dcae] Google Protocol Buffer support for VES

2017-07-14 Thread Pasi Vaananen 
Hi Manoj,

Thank you for the pointers. I looked on the performance test info on the
slides, and in addition of them being quite old & conducted in low end
HW, it is hard to tell what the messaging performance was from the
"query a list of course numbers" (?) test scenario. I am sure that we
can find more recent info somewhere, at least for GPB.

I am aware of VES work and specs, including the efforts to try it out in
the context of infrastructure events and metrics around OPNFV Barometer
project - which was why I was asking on the context that you had in
mind. From our perspective, less protocols for the same / similar
purposes would be good target, provided that we could get to something
that meets the associated requirements.

Pasi


On 07/14/2017 10:56 AM, Manoj K Nair wrote:
>
> Hi Pasi,
>
>  
>
> Our suggestion was specifically for VNF Event Streaming
> (https://wiki.opnfv.org/display/ves/VES+Home) which is one of the
> options supported as collection mechanism in DCAE.  Please refer to
> the documents shared by Alok here
> <https://wiki.onap.org/display/DW/VNF+Requirement+validation+for+DCAE>
> . Regarding your comment on performance – please refer to an
> evaluation available in public domain here
> <https://www.slideshare.net/IgorAnishchenko/pb-vs-thrift-vs-avro>
>  which compares Protobuf, Avro and Thrift.
>
>  
>
> Protobuf can define the schema (VNF originated event format) in a
> .proto file. Example can be seen here
> <https://blogs.cisco.com/sp/streaming-telemetry-with-google-protocol-buffers>.
>
>
>  
>
> Regards
>
>  
>
> Manoj
>
>  
>
>  
>
>  
>
> *From:*onap-discuss-boun...@lists.onap.org
> [mailto:onap-discuss-boun...@lists.onap.org] *On Behalf Of *Pasi Vaananen
> *Sent:* Friday, July 14, 2017 6:51 PM
> *To:* onap-discuss@lists.onap.org
> *Subject:* Re: [onap-discuss] [dcae] Google Protocol Buffer support
> for VES
>
>  
>
> [External Email]
>
> 
>
>  
>
> Hi Manoj,
>
> Is your proposal in any way aligned with what has been proposed on
> openconfig <http://www.openconfig.net/> streaming telemetry project,
> or would this be an independent ONAP specific version built around the
> gRPC ?
>
> Would this be used just for the VNF originated events/metrics, or also
> for the infrastructure originated events/metrics ?
>
> Would you tie the descriptions of VNF originated events to some way to
> describe what VNF is able to expose as part of the VNF package (either
> tied to TOSCA or some other way) ?
>
> We have been testing the capabilities of Kafka vs. AMQP (using Apache
> QPID dispatch router
> <https://qpid.apache.org/components/dispatch-router/index.html>) with
> the very specific goal to assess delivery capabilities of low-latency
> events (targeting fault events that need "fast", sub-50ms remediation
> processes) along with large amounts of metrics data in the same
> messaging "bus". We have not tested either VES or gRPC performance,
> but binary encoding could also help on the events side, as it could
> eliminate the unnecessary encoding/parsing steps from the path
> (obviously assuming that the event syntax, semantics and context data
> are well defined in the first place).
>
> Thanks,
>
> Pasi Vaananen,
>
> Systems Architect, NFV
>
> Office of Technology, Red Hat
>
>  
>
> On 07/14/2017 05:01 AM, Manoj K Nair wrote:
>
> Hi Alok,
>
>  
>
> We reviewed the VES specification you shared after the
> presentation in DCAE weekly meeting couple of weeks back.  We
> understand that currently VES supports a REST based interface with
> JSON payload to push telemetry data and events from the VNF to
> DCAE VES collector. For telemetry control an HTTP response channel
> is being used as per the test code link you published in the
> presentation.
>
>  
>
> Just wondering if GPB is considered as one of potential options
> for VES.  You have noted the use of Avro as one potential option
> as future enhancement. Wondering if GPB is also a potential option
> you are considering. Some of the advantages we see for GPB are.
>
> •   Uses binary formatting during serialization, which densily
> packs the payload (which is good for PM/telemetry data)
>
> •   Faster than the VES proposed JSON payload (which is good
> for events and alarms)
>
> •   Can define the structure (schema) of events/telemetry data
> in a protocol definition file (.proto) which is similar to the
> JSON schema that is available in OPNFV VES project
>
>

Re: [onap-discuss] [dcae] Google Protocol Buffer support for VES

2017-07-14 Thread Manoj K Nair 
Hi Pasi,

Our suggestion was specifically for VNF Event Streaming 
(https://wiki.opnfv.org/display/ves/VES+Home) which is one of the options 
supported as collection mechanism in DCAE.  Please refer to the documents 
shared by Alok 
here<https://wiki.onap.org/display/DW/VNF+Requirement+validation+for+DCAE> . 
Regarding your comment on performance – please refer to an evaluation available 
in public domain 
here<https://www.slideshare.net/IgorAnishchenko/pb-vs-thrift-vs-avro>  which 
compares Protobuf, Avro and Thrift.

Protobuf can define the schema (VNF originated event format) in a .proto file. 
Example can be seen 
here<https://blogs.cisco.com/sp/streaming-telemetry-with-google-protocol-buffers>.

Regards

Manoj



From: onap-discuss-boun...@lists.onap.org 
[mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of Pasi Vaananen
Sent: Friday, July 14, 2017 6:51 PM
To: onap-discuss@lists.onap.org
Subject: Re: [onap-discuss] [dcae] Google Protocol Buffer support for VES


[External Email]




Hi Manoj,

Is your proposal in any way aligned with what has been proposed on 
openconfig<http://www.openconfig.net/> streaming telemetry project, or would 
this be an independent ONAP specific version built around the gRPC ?

Would this be used just for the VNF originated events/metrics, or also for the 
infrastructure originated events/metrics ?

Would you tie the descriptions of VNF originated events to some way to describe 
what VNF is able to expose as part of the VNF package (either tied to TOSCA or 
some other way) ?

We have been testing the capabilities of Kafka vs. AMQP (using Apache QPID 
dispatch router<https://qpid.apache.org/components/dispatch-router/index.html>) 
with the very specific goal to assess delivery capabilities of low-latency 
events (targeting fault events that need "fast", sub-50ms remediation 
processes) along with large amounts of metrics data in the same messaging 
"bus". We have not tested either VES or gRPC performance, but binary encoding 
could also help on the events side, as it could eliminate the unnecessary 
encoding/parsing steps from the path (obviously assuming that the event syntax, 
semantics and context data are well defined in the first place).

Thanks,

Pasi Vaananen,

Systems Architect, NFV

Office of Technology, Red Hat

On 07/14/2017 05:01 AM, Manoj K Nair wrote:
Hi Alok,

We reviewed the VES specification you shared after the presentation in DCAE 
weekly meeting couple of weeks back.  We understand that currently VES supports 
a REST based interface with JSON payload to push telemetry data and events from 
the VNF to DCAE VES collector. For telemetry control an HTTP response channel 
is being used as per the test code link you published in the presentation.

Just wondering if GPB is considered as one of potential options for VES.  You 
have noted the use of Avro as one potential option as future enhancement. 
Wondering if GPB is also a potential option you are considering. Some of the 
advantages we see for GPB are.
•   Uses binary formatting during serialization, which densily packs the 
payload (which is good for PM/telemetry data)
•   Faster than the VES proposed JSON payload (which is good for events and 
alarms)
•   Can define the structure (schema) of events/telemetry data in a 
protocol definition file (.proto) which is similar to the JSON schema that is 
available in OPNFV VES project
•   Availability of code generators in multiple programming languages for 
encoding and decoding - Can generate code in C#, C++,  Java, Python, Go . 
Options in other languages available as well.

While we have noted the dynamic schema capabilities in Avro, one key advantage 
we see with GPB is the serialization code generation in multiple languages 
without limiting to the C language based library which is currently available 
and open source tool sets available for GPB. Moreover many network vendors and 
open source monitoring solutions (e.g. Prometheus, 
link<https://prometheus.io/docs/instrumenting/exposition_formats/>)   support 
GPB already. We would be glad to share more information on the potential GPB 
usage if required.

Regards

Manoj
[https://www.netcracker.com/assets/img/netcracker-social-final.png]ƕ




The information transmitted herein is intended only for the person or entity to 
which it is addressed and may contain confidential, proprietary and/or 
privileged material. Any review, retransmission, dissemination or other use of, 
or taking of any action in reliance upon, this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and delete the material from any computer.



___

onap-discuss mailing list

onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap

Re: [onap-discuss] [dcae] Google Protocol Buffer support for VES

2017-07-14 Thread GUPTA, ALOK 
Manoj:

We are trying to go slow. We have not finalized on the protocol to use. We can 
certainly consider using GPB. I have forwarded your email to my architects for 
their input. I will keep you updated. BTW we also have a JAVA library 
completed. It will take some time to show up in ONAP gerrit (need to go through 
 the process),  but you can pick it up from our github: 
https://github.com/att/evel-library/files/1145800/VESJavaLibrary.tar.gz


Regards,

Alok Gupta
732-420-7007
MT B2 3D30
ag1...@att.com

From: Manoj K Nair [mailto:manoj.k.n...@netcracker.com]
Sent: Friday, July 14, 2017 5:01 AM
To: GUPTA, ALOK 
Cc: onap-discuss@lists.onap.org
Subject: [dcae] Google Protocol Buffer support for VES

Hi Alok,

We reviewed the VES specification you shared after the presentation in DCAE 
weekly meeting couple of weeks back.  We understand that currently VES supports 
a REST based interface with JSON payload to push telemetry data and events from 
the VNF to DCAE VES collector. For telemetry control an HTTP response channel 
is being used as per the test code link you published in the presentation.

Just wondering if GPB is considered as one of potential options for VES.  You 
have noted the use of Avro as one potential option as future enhancement. 
Wondering if GPB is also a potential option you are considering. Some of the 
advantages we see for GPB are.
•Uses binary formatting during serialization, which densily packs the 
payload (which is good for PM/telemetry data)
•Faster than the VES proposed JSON payload (which is good for events 
and alarms)
•Can define the structure (schema) of events/telemetry data in a 
protocol definition file (.proto) which is similar to the JSON schema that is 
available in OPNFV VES project
•Availability of code generators in multiple programming languages for 
encoding and decoding - Can generate code in C#, C++,  Java, Python, Go . 
Options in other languages available as well.

While we have noted the dynamic schema capabilities in Avro, one key advantage 
we see with GPB is the serialization code generation in multiple languages 
without limiting to the C language based library which is currently available 
and open source tool sets available for GPB. Moreover many network vendors and 
open source monitoring solutions (e.g. Prometheus, 
link)
   support GPB already. We would be glad to share more information on the 
potential GPB usage if required.

Regards

Manoj
[cid:image002.png@01D2FC7C.E35B2030][https://www.netcracker.com/assets/img/netcracker-social-final.png]ƕ




The information transmitted herein is intended only for the person or entity to 
which it is addressed and may contain confidential, proprietary and/or 
privileged material. Any review, retransmission, dissemination or other use of, 
or taking of any action in reliance upon, this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and delete the material from any computer.
___
onap-discuss mailing list
onap-discuss@lists.onap.org
https://lists.onap.org/mailman/listinfo/onap-discuss

[onap-discuss] [dcae] Google Protocol Buffer support for VES

2017-07-14 Thread Manoj K Nair 
Hi Alok,

We reviewed the VES specification you shared after the presentation in DCAE 
weekly meeting couple of weeks back.  We understand that currently VES supports 
a REST based interface with JSON payload to push telemetry data and events from 
the VNF to DCAE VES collector. For telemetry control an HTTP response channel 
is being used as per the test code link you published in the presentation.

Just wondering if GPB is considered as one of potential options for VES.  You 
have noted the use of Avro as one potential option as future enhancement. 
Wondering if GPB is also a potential option you are considering. Some of the 
advantages we see for GPB are.
•   Uses binary formatting during serialization, which densily packs the 
payload (which is good for PM/telemetry data)
•   Faster than the VES proposed JSON payload (which is good for events and 
alarms)
•   Can define the structure (schema) of events/telemetry data in a 
protocol definition file (.proto) which is similar to the JSON schema that is 
available in OPNFV VES project
•   Availability of code generators in multiple programming languages for 
encoding and decoding - Can generate code in C#, C++,  Java, Python, Go . 
Options in other languages available as well.

While we have noted the dynamic schema capabilities in Avro, one key advantage 
we see with GPB is the serialization code generation in multiple languages 
without limiting to the C language based library which is currently available 
and open source tool sets available for GPB. Moreover many network vendors and 
open source monitoring solutions (e.g. Prometheus, 
link)   support 
GPB already. We would be glad to share more information on the potential GPB 
usage if required.

Regards

Manoj
[https://www.netcracker.com/assets/img/netcracker-social-final.png] ƕ




The information transmitted herein is intended only for the person or entity to 
which it is addressed and may contain confidential, proprietary and/or 
privileged material. Any review, retransmission, dissemination or other use of, 
or taking of any action in reliance upon, this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and delete the material from any computer.
___
onap-discuss mailing list
onap-discuss@lists.onap.org
https://lists.onap.org/mailman/listinfo/onap-discuss