Hi Alan,
On Jun 29, 2011, at 12:26 AM, Zhongjie Wang wrote:
> On Wed, Jun 29, 2011 at 10:02 AM, Adriano Monteiro Marques
> <[email protected]> wrote:
> Sounds good to me... any objections Diogo, Alan?
>
> On Jun 28, 2011, at 2:59 PM, Zubair Nabi wrote:
>
>> Guys,
>>
>> I propose that we add a field: required string testType
>> to message Test to differentiate between a website test and a service test.
>> Similar to the field in message Events.
>>
>> Also, Alan you haven't added the message that will encapsulate all other
>> messages for p2p communication to the common proto. Can you please do so
>> now? Or we still haven't decided that this is the best approach? Because it
>> has my vote. Like you said, if a message doesn't exist, it will not take any
>> space.
>
> Yes, I agree.
> I just didn't get the agreement from Adriano to change the protocol messages.
> Shall I modify it?
Sure, if you, Diogo and Zubair are in agreement, go ahead and change everything
as necessary.
> Now I'm using the preceding name string before the message.
> The format:
> TotalMsgLength NameLength Name ProtobufMsgLength
> SerializedProtobufMessage
> 4 bytes 2 bytes n bytes 2 bytes
> m bytes
>
> BTW, I suggest we send a 2 or 4 bytes length before sending strings,
> according to the
> DER(http://en.wikipedia.org/wiki/Distinguished_Encoding_Rules) by ITU.
Is that what the aggregator and the mobile agents expecting from you, or is
this something new?
>
>>
>> On Fri, Jun 10, 2011 at 8:13 PM, Zhongjie Wang <[email protected]> wrote:
>> Hi Adriano,
>>
>> On Fri, Jun 10, 2011 at 11:03 PM, Adriano Monteiro Marques
>> <[email protected]> wrote:
>> Hi,
>>
>> On Jun 10, 2011, at 11:34 AM, Zhongjie Wang wrote:
>>
>>> Hi Guys,
>>>
>>> On Fri, Jun 10, 2011 at 10:02 PM, Zubair Nabi <[email protected]>
>>> wrote:
>>> Hi Guys,
>>>
>>>
>>>> If the token is sent with the message in plain text, then someone may
>>>> perform a wire-tapping attack.
>>>> And if the two party in communication has already had a handshake progress
>>>> and figured out a symmetric key
>>>> for current session, then we don't need the token i guess, the symmetric
>>>> key itself is enough for the encryption and authentication purpose.
>>>> It's OK to keep agentID as you said.
>>>> Hm... Zubair helped me to set that there, let's just confirm with him that
>>>> we don't need it there. Your consideration is fair, but I'm not sure this
>>>> is the only purpose the token is there.
>>>
>>> The token is also supposed to reflect the state of the agent binary. For
>>> example, if the binary is updated, the symmetric key would remain the same
>>> but the token would change.
>>> And we aren't going to send messages in plain text. We're going to encrypt
>>> each message.
>>>
>>> Yes, in the Event message we don't need to send back the unnecessary
>>> fields.
>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional" in
>>> the Detail structure,
>>> they can be omitted while sending.
>>> Because when we send back Detail in Event message, we need to clone it and
>>> only send the necessary fields.
>>> Ok, sounds good. Can you get them fixed, and let the others know about your
>>> fix?
>>> +1
>>>
>>> So that's what I propose at the beginning.
>>> All messages will be instantiate as a common message type.
>>> It has a type field and some option fields.
>>> The receiver will instantiate it as a common ICMMessage for example,
>>> then read the type field, and choose which optional field to read.
>>> The optional fields which doesn't exists in a message don't take any spaces.
>>> e.g.
>>> message ICMMessage {
>>> required int64 agentID = 1;
>>> required int32 type = 2;
>>> optional AssignTask assignTask = 100;
>>> optional UpgradeToSuper upgradeToSuper = 101;
>>> optional VersionUpdate versionUpdate = 102;
>>> // extensions 100 to max;
>>> }
>>> Did you try this approach? Not sure it will work with protocol buffers.
>>>
>>> Luis and I discussed this a while back and we came to the conclusion that
>>> this won't be work. In case of p2p communication, we will have to send an
>>> additional message.
>>> For example our send would look something like send(string type, string
>>> message)
>>> At the receiving end, this "type" would tell us which type of Protobuf
>>> message this is.
>>>
>>> Hi Zubair, I'm using the way you said now. But I think what I mentioned
>>> also works. I just tested that. ;)
>>> Here's the .proto file and test.py I used.
>>
>> Oh, now I understood your approach. We would have one type of message that
>> would encapsulate all others, right?
>> The problem with this approach is that the message gets to big and slower to
>> process. Though it works, protobuf is got to parse the whole thing, and it
>> will place several markers in the message to indicate that the field is
>> empty. This can escalate to a few hundreds easily as we extend our app.
>>
>>
>> I was worrying about the message size. But after tested, I found even I
>> added more optional fields to that message. The size didn't grow, as I have
>> printed out. (I added ExtendedMessageType up to 10).
>> I think it's because the protobuf message use unique tag number to indicate
>> the field.
>> If an optional field doesn't exist, it will not take any space in the
>> message.
>>
>>>
>>> test.proto:
>>>
>>> // this is a sample message sent from the aggregator to the agent
>>> message ICMMessage {
>>> required int32 type = 1;
>>>
>>> optional AssignTask assignTask = 100;
>>> optional UpgradeToSuper upgradeToSuper = 101;
>>> optional ExtendedMessageType1 em1 = 102;
>>> optional ExtendedMessageType2 em2 = 103;
>>> optional ExtendedMessageType3 em3 = 104;
>>> // extensions 100 to max;
>>> }
>>>
>>> message AssignTask {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message UpgradeToSuper {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message ExtendedMessageType1 {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message ExtendedMessageType2 {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message ExtendedMessageType3 {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>>
>>> test.py:
>>>
>>> import test_pb2
>>>
>>> MSG_ASSIGN_TASK = 1
>>> MSG_UPGRADE_TO_SUPER = 2
>>> MessageTypes = {1:'AssignTask', 2:'UpgradeToSuper'}
>>>
>>> msg = test_pb2.ICMMessage()
>>> msg.type = MSG_ASSIGN_TASK
>>> msg.assignTask.foo = 11
>>> msg.assignTask.bar = 22
>>> msg_send = msg.SerializeToString()
>>>
>>> print(msg)
>>> print(msg_send)
>>> print("Length: %d" % len(msg_send))
>>>
>>> msg_recv = msg_send
>>>
>>> msg1 = test_pb2.ICMMessage()
>>> msg1.ParseFromString(msg_recv)
>>> print("Got a message. Type: %s" % MessageTypes[msg1.type])
>>> print("Detail:")
>>> print(msg1.assignTask)
>>>
>>> The result:
>>>
>>> <image.png>
>>>
>>>
>>>
>>> On Fri, Jun 10, 2011 at 6:49 PM, Adriano Monteiro Marques
>>> <[email protected]> wrote:
>>> Hi Alan,
>>>
>>> On Jun 10, 2011, at 12:53 AM, Zhongjie Wang wrote:
>>>
>>>> Hi Adriano,
>>>> Thanks for the timely response. :)
>>>>
>>>> On Thu, Jun 9, 2011 at 10:44 PM, Adriano Monteiro Marques
>>>> <[email protected]> wrote:
>>>> Hi Alan,
>>>>
>>>> On Jun 9, 2011, at 3:01 AM, Zhongjie Wang wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> About the message format, I have some comments.
>>>>> 1. Seems there're duplicate fields in SendWebsiteReport and
>>>>> SendServiceReport:
>>>>> 162
>>>>> // send_report
>>>>> 163
>>>>> message SendWebsiteReport {
>>>>> 164
>>>>> required RequestHeader header = 1;
>>>>> 165
>>>>> required WebsiteReport report = 2;
>>>>> 166
>>>>> }
>>>>> In the RequestHeader structure to the aggregator, there're are
>>>>> token and agentID. And in WebsiteReport.ICMReport, there're also agentID.
>>>>> Though I didn't know the difference between token and agentID. I suggest
>>>>> we eliminate the agentID field in ICMReport.
>>>>
>>>> Token is our to confirm to the aggregator that we are legitimate. On the
>>>> duplicated agentID, it is fine: If a node is passing a website report from
>>>> another node, then the ICMReport will show the agentID for the origin
>>>> node, and the RequestHeader will provide the passing agent's id.
>>>>
>>>>
>>>> If the token is sent with the message in plain text, then someone may
>>>> perform a wire-tapping attack.
>>>> And if the two party in communication has already had a handshake progress
>>>> and figured out a symmetric key
>>>> for current session, then we don't need the token i guess, the symmetric
>>>> key itself is enough for the encryption and authentication purpose.
>>>> It's OK to keep agentID as you said.
>>>
>>> Hm... Zubair helped me to set that there, let's just confirm with him that
>>> we don't need it there. Your consideration is fair, but I'm not sure this
>>> is the only purpose the token is there.
>>>
>>>>
>>>>> 2. Besides that, the optional fields: redirectLink, htmlResponse,
>>>>> htmlMedia, seems to be better in the WebsiteReportDetail structure,
>>>>> because it's more test specific. I suppose the Report messages should
>>>>> have only two parts, the header and the detail.
>>>>
>>>> Check the Event message... when we send back a WebsiteReportDetail or
>>>> ServiceReportDetail we don't send out those details as they're not really
>>>> necessary.
>>>>
>>>>
>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>> fields.
>>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional" in
>>>> the Detail structure,
>>>> they can be omitted while sending.
>>>> Because when we send back Detail in Event message, we need to clone it and
>>>> only send the necessary fields.
>>>
>>> Ok, sounds good. Can you get them fixed, and let the others know about your
>>> fix?
>>>
>>>>
>>>>> 3. Since we don't have a field in the message to identify the
>>>>> message type, I don't know how we can create a message instance after
>>>>> receiving a bunch of raw bytes. Currently I send the message type string
>>>>> before send the message itself. But I think the better solution is tag it
>>>>> in the message structure.
>>>>
>>>> Even if we had a field in the message, how would we retrieve it to know
>>>> the type prior to instantiating it? What we're doing on the Aggregator
>>>> side is to presume the sort of message a given API is supposed to
>>>> send/receive. Perhaps we could do the same with the desktop agent, what do
>>>> you think?
>>>>
>>>>
>>>> So that's what I propose at the beginning.
>>>> All messages will be instantiate as a common message type.
>>>> It has a type field and some option fields.
>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>> then read the type field, and choose which optional field to read.
>>>> The optional fields which doesn't exists in a message don't take any
>>>> spaces.
>>>> e.g.
>>>> message ICMMessage {
>>>> required int64 agentID = 1;
>>>> required int32 type = 2;
>>>>
>>>> optional AssignTask assignTask = 100;
>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>> optional VersionUpdate versionUpdate = 102;
>>>> // extensions 100 to max;
>>>> }
>>>
>>> Did you try this approach? Not sure it will work with protocol buffers.
>>>
>>>>
>>>>> 4. Another slightly modification, I suggest that we use float type
>>>>> for responseTime, the unit is seconds, for accuracy.
>>>>
>>>> We're using miliseconds there as mentioned in a previous email, that's the
>>>> reason why we use integer. It saves on message size and keeps accuracy.
>>>>
>>>> Well noticed Alan! Keep pushing! Let me know if there still is anything
>>>> you feel like it is wrong or that is stopping you from progressing.
>>>>
>>>>
>>>> Thanks, Adriano! I'll keep on thinking and make it perfect. :)
>>>>
>>>>
>>>> Cheers!
>>>>
>>>>>
>>>>> Best Regards.
>>>>> Alan
>>>>>
>>>>> On Sat, Jun 4, 2011 at 5:34 PM, Adriano Monteiro Marques
>>>>> <[email protected]> wrote:
>>>>> Right, mainly for nodes who stays connected for long periods of time.
>>>>> Also, if we want a test to be run ASAP, we need to send this info
>>>>> everytime so the node knows of a brand new test version, updates their
>>>>> test set and run them ASAP. On the software version, if we figure any
>>>>> security issue, the nodes will also update ASAP.
>>>>>
>>>>>
>>>>> On Jun 4, 2011, at 6:22 AM, Zubair Nabi wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I think the idea was to keep the binary and test versions as fresh as
>>>>>> possible. This ResponseHeader would make sure of that.
>>>>>>
>>>>>> On Sat, Jun 4, 2011 at 4:04 AM, Diogo Pinheiro
>>>>>> <[email protected]> wrote:
>>>>>> Hi
>>>>>>
>>>>>> I have another question. Why the ResponseHeader has the currentVersionNo
>>>>>> and currentTestsVersion ? I think we shouldn't send that information all
>>>>>> the time. Just when the agent connect, and in a rpc call.
>>>>>>
>>>>>> What do you think ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best,
>>>>>> __
>>>>>> Zubair
>>>>>
>>>>> ---
>>>>> Adriano Monteiro Marques
>>>>>
>>>>> http://www.thoughtspad.com
>>>>> http://www.umitproject.org
>>>>> http://blog.umitproject.org
>>>>> http://www.pythonbenelux.org
>>>>>
>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Simplify data backup and recovery for your virtual environment with
>>>>> vRanger.
>>>>> Installation's a snap, and flexible recovery options mean your data is
>>>>> safe,
>>>>> secure and there when you need it. Discover what all the cheering's about.
>>>>> Get your free trial download today.
>>>>> http://p.sf.net/sfu/quest-dev2dev2
>>>>> _______________________________________________
>>>>> Umit-devel mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhongjie Wang
>>>>> Master Candidate
>>>>> Computer System Architecture
>>>>> Peking University, China
>>>>
>>>>
>>>> ---
>>>> Adriano Monteiro Marques
>>>>
>>>> http://www.thoughtspad.com
>>>> http://www.umitproject.org
>>>> http://blog.umitproject.org
>>>> http://www.pythonbenelux.org
>>>>
>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhongjie Wang
>>>> Master Candidate
>>>> Computer System Architecture
>>>> Peking University, China
>>>
>>>
>>> ---
>>> Adriano Monteiro Marques
>>>
>>> http://www.thoughtspad.com
>>> http://www.umitproject.org
>>> http://blog.umitproject.org
>>> http://www.pythonbenelux.org
>>>
>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>
>>>
>>>
>>>
>>> --
>>> Best,
>>> __
>>> Zubair
>>>
>>>
>>>
>>> --
>>> Zhongjie Wang
>>> Master Candidate
>>> Computer System Architecture
>>> Peking University, China
>>
>>
>> ---
>> Adriano Monteiro Marques
>>
>> http://www.thoughtspad.com
>> http://www.umitproject.org
>> http://blog.umitproject.org
>> http://www.pythonbenelux.org
>>
>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>
>>
>>
>>
>> --
>> Zhongjie Wang
>> Master Candidate
>> Computer System Architecture
>> Peking University, China
>>
>>
>>
>> --
>> Best,
>> __
>> Zubair
>
> ---
> Adriano Monteiro Marques
>
> http://www.thoughtspad.com
> http://www.umitproject.org
> http://blog.umitproject.org
> http://www.pythonbenelux.org
>
> "Don't stay in bed, unless you can make money in bed." - George Burns
>
>
>
>
> --
> Zhongjie Wang
> Master Candidate
> Computer System Architecture
> Peking University, China
---
Adriano Monteiro Marques
http://www.thoughtspad.com
http://www.umitproject.org
http://blog.umitproject.org
http://www.pythonbenelux.org
"Don't stay in bed, unless you can make money in bed." - George Burns
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Umit-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/umit-devel
