Sounds good to me... any objections Diogo, Alan?
On Jun 28, 2011, at 2:59 PM, Zubair Nabi wrote:
> Guys,
>
> I propose that we add a field: required string testType
> to message Test to differentiate between a website test and a service test.
> Similar to the field in message Events.
>
> Also, Alan you haven't added the message that will encapsulate all other
> messages for p2p communication to the common proto. Can you please do so now?
> Or we still haven't decided that this is the best approach? Because it has my
> vote. Like you said, if a message doesn't exist, it will not take any space.
>
> On Fri, Jun 10, 2011 at 8:13 PM, Zhongjie Wang <[email protected]> wrote:
> Hi Adriano,
>
> On Fri, Jun 10, 2011 at 11:03 PM, Adriano Monteiro Marques
> <[email protected]> wrote:
> Hi,
>
> On Jun 10, 2011, at 11:34 AM, Zhongjie Wang wrote:
>
>> Hi Guys,
>>
>> On Fri, Jun 10, 2011 at 10:02 PM, Zubair Nabi <[email protected]>
>> wrote:
>> Hi Guys,
>>
>>
>>> If the token is sent with the message in plain text, then someone may
>>> perform a wire-tapping attack.
>>> And if the two party in communication has already had a handshake progress
>>> and figured out a symmetric key
>>> for current session, then we don't need the token i guess, the symmetric
>>> key itself is enough for the encryption and authentication purpose.
>>> It's OK to keep agentID as you said.
>>> Hm... Zubair helped me to set that there, let's just confirm with him that
>>> we don't need it there. Your consideration is fair, but I'm not sure this
>>> is the only purpose the token is there.
>>
>> The token is also supposed to reflect the state of the agent binary. For
>> example, if the binary is updated, the symmetric key would remain the same
>> but the token would change.
>> And we aren't going to send messages in plain text. We're going to encrypt
>> each message.
>>
>> Yes, in the Event message we don't need to send back the unnecessary fields.
>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional" in
>> the Detail structure,
>> they can be omitted while sending.
>> Because when we send back Detail in Event message, we need to clone it and
>> only send the necessary fields.
>> Ok, sounds good. Can you get them fixed, and let the others know about your
>> fix?
>> +1
>>
>> So that's what I propose at the beginning.
>> All messages will be instantiate as a common message type.
>> It has a type field and some option fields.
>> The receiver will instantiate it as a common ICMMessage for example,
>> then read the type field, and choose which optional field to read.
>> The optional fields which doesn't exists in a message don't take any spaces.
>> e.g.
>> message ICMMessage {
>> required int64 agentID = 1;
>> required int32 type = 2;
>> optional AssignTask assignTask = 100;
>> optional UpgradeToSuper upgradeToSuper = 101;
>> optional VersionUpdate versionUpdate = 102;
>> // extensions 100 to max;
>> }
>> Did you try this approach? Not sure it will work with protocol buffers.
>>
>> Luis and I discussed this a while back and we came to the conclusion that
>> this won't be work. In case of p2p communication, we will have to send an
>> additional message.
>> For example our send would look something like send(string type, string
>> message)
>> At the receiving end, this "type" would tell us which type of Protobuf
>> message this is.
>>
>> Hi Zubair, I'm using the way you said now. But I think what I mentioned also
>> works. I just tested that. ;)
>> Here's the .proto file and test.py I used.
>
> Oh, now I understood your approach. We would have one type of message that
> would encapsulate all others, right?
> The problem with this approach is that the message gets to big and slower to
> process. Though it works, protobuf is got to parse the whole thing, and it
> will place several markers in the message to indicate that the field is
> empty. This can escalate to a few hundreds easily as we extend our app.
>
>
> I was worrying about the message size. But after tested, I found even I added
> more optional fields to that message. The size didn't grow, as I have printed
> out. (I added ExtendedMessageType up to 10).
> I think it's because the protobuf message use unique tag number to indicate
> the field.
> If an optional field doesn't exist, it will not take any space in the message.
>
>>
>> test.proto:
>>
>> // this is a sample message sent from the aggregator to the agent
>> message ICMMessage {
>> required int32 type = 1;
>>
>> optional AssignTask assignTask = 100;
>> optional UpgradeToSuper upgradeToSuper = 101;
>> optional ExtendedMessageType1 em1 = 102;
>> optional ExtendedMessageType2 em2 = 103;
>> optional ExtendedMessageType3 em3 = 104;
>> // extensions 100 to max;
>> }
>>
>> message AssignTask {
>> required int32 foo = 1;
>> required int32 bar = 2;
>> }
>>
>> message UpgradeToSuper {
>> required int32 foo = 1;
>> required int32 bar = 2;
>> }
>>
>> message ExtendedMessageType1 {
>> required int32 foo = 1;
>> required int32 bar = 2;
>> }
>>
>> message ExtendedMessageType2 {
>> required int32 foo = 1;
>> required int32 bar = 2;
>> }
>>
>> message ExtendedMessageType3 {
>> required int32 foo = 1;
>> required int32 bar = 2;
>> }
>>
>>
>> test.py:
>>
>> import test_pb2
>>
>> MSG_ASSIGN_TASK = 1
>> MSG_UPGRADE_TO_SUPER = 2
>> MessageTypes = {1:'AssignTask', 2:'UpgradeToSuper'}
>>
>> msg = test_pb2.ICMMessage()
>> msg.type = MSG_ASSIGN_TASK
>> msg.assignTask.foo = 11
>> msg.assignTask.bar = 22
>> msg_send = msg.SerializeToString()
>>
>> print(msg)
>> print(msg_send)
>> print("Length: %d" % len(msg_send))
>>
>> msg_recv = msg_send
>>
>> msg1 = test_pb2.ICMMessage()
>> msg1.ParseFromString(msg_recv)
>> print("Got a message. Type: %s" % MessageTypes[msg1.type])
>> print("Detail:")
>> print(msg1.assignTask)
>>
>> The result:
>>
>> <image.png>
>>
>>
>>
>> On Fri, Jun 10, 2011 at 6:49 PM, Adriano Monteiro Marques
>> <[email protected]> wrote:
>> Hi Alan,
>>
>> On Jun 10, 2011, at 12:53 AM, Zhongjie Wang wrote:
>>
>>> Hi Adriano,
>>> Thanks for the timely response. :)
>>>
>>> On Thu, Jun 9, 2011 at 10:44 PM, Adriano Monteiro Marques
>>> <[email protected]> wrote:
>>> Hi Alan,
>>>
>>> On Jun 9, 2011, at 3:01 AM, Zhongjie Wang wrote:
>>>
>>>> Hi all,
>>>>
>>>> About the message format, I have some comments.
>>>> 1. Seems there're duplicate fields in SendWebsiteReport and
>>>> SendServiceReport:
>>>> 162
>>>> // send_report
>>>> 163
>>>> message SendWebsiteReport {
>>>> 164
>>>> required RequestHeader header = 1;
>>>> 165
>>>> required WebsiteReport report = 2;
>>>> 166
>>>> }
>>>> In the RequestHeader structure to the aggregator, there're are
>>>> token and agentID. And in WebsiteReport.ICMReport, there're also agentID.
>>>> Though I didn't know the difference between token and agentID. I suggest
>>>> we eliminate the agentID field in ICMReport.
>>>
>>> Token is our to confirm to the aggregator that we are legitimate. On the
>>> duplicated agentID, it is fine: If a node is passing a website report from
>>> another node, then the ICMReport will show the agentID for the origin node,
>>> and the RequestHeader will provide the passing agent's id.
>>>
>>>
>>> If the token is sent with the message in plain text, then someone may
>>> perform a wire-tapping attack.
>>> And if the two party in communication has already had a handshake progress
>>> and figured out a symmetric key
>>> for current session, then we don't need the token i guess, the symmetric
>>> key itself is enough for the encryption and authentication purpose.
>>> It's OK to keep agentID as you said.
>>
>> Hm... Zubair helped me to set that there, let's just confirm with him that
>> we don't need it there. Your consideration is fair, but I'm not sure this is
>> the only purpose the token is there.
>>
>>>
>>>> 2. Besides that, the optional fields: redirectLink, htmlResponse,
>>>> htmlMedia, seems to be better in the WebsiteReportDetail structure,
>>>> because it's more test specific. I suppose the Report messages should have
>>>> only two parts, the header and the detail.
>>>
>>> Check the Event message... when we send back a WebsiteReportDetail or
>>> ServiceReportDetail we don't send out those details as they're not really
>>> necessary.
>>>
>>>
>>> Yes, in the Event message we don't need to send back the unnecessary
>>> fields.
>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional" in
>>> the Detail structure,
>>> they can be omitted while sending.
>>> Because when we send back Detail in Event message, we need to clone it and
>>> only send the necessary fields.
>>
>> Ok, sounds good. Can you get them fixed, and let the others know about your
>> fix?
>>
>>>
>>>> 3. Since we don't have a field in the message to identify the
>>>> message type, I don't know how we can create a message instance after
>>>> receiving a bunch of raw bytes. Currently I send the message type string
>>>> before send the message itself. But I think the better solution is tag it
>>>> in the message structure.
>>>
>>> Even if we had a field in the message, how would we retrieve it to know the
>>> type prior to instantiating it? What we're doing on the Aggregator side is
>>> to presume the sort of message a given API is supposed to send/receive.
>>> Perhaps we could do the same with the desktop agent, what do you think?
>>>
>>>
>>> So that's what I propose at the beginning.
>>> All messages will be instantiate as a common message type.
>>> It has a type field and some option fields.
>>> The receiver will instantiate it as a common ICMMessage for example,
>>> then read the type field, and choose which optional field to read.
>>> The optional fields which doesn't exists in a message don't take any spaces.
>>> e.g.
>>> message ICMMessage {
>>> required int64 agentID = 1;
>>> required int32 type = 2;
>>>
>>> optional AssignTask assignTask = 100;
>>> optional UpgradeToSuper upgradeToSuper = 101;
>>> optional VersionUpdate versionUpdate = 102;
>>> // extensions 100 to max;
>>> }
>>
>> Did you try this approach? Not sure it will work with protocol buffers.
>>
>>>
>>>> 4. Another slightly modification, I suggest that we use float type
>>>> for responseTime, the unit is seconds, for accuracy.
>>>
>>> We're using miliseconds there as mentioned in a previous email, that's the
>>> reason why we use integer. It saves on message size and keeps accuracy.
>>>
>>> Well noticed Alan! Keep pushing! Let me know if there still is anything you
>>> feel like it is wrong or that is stopping you from progressing.
>>>
>>>
>>> Thanks, Adriano! I'll keep on thinking and make it perfect. :)
>>>
>>>
>>> Cheers!
>>>
>>>>
>>>> Best Regards.
>>>> Alan
>>>>
>>>> On Sat, Jun 4, 2011 at 5:34 PM, Adriano Monteiro Marques
>>>> <[email protected]> wrote:
>>>> Right, mainly for nodes who stays connected for long periods of time.
>>>> Also, if we want a test to be run ASAP, we need to send this info
>>>> everytime so the node knows of a brand new test version, updates their
>>>> test set and run them ASAP. On the software version, if we figure any
>>>> security issue, the nodes will also update ASAP.
>>>>
>>>>
>>>> On Jun 4, 2011, at 6:22 AM, Zubair Nabi wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I think the idea was to keep the binary and test versions as fresh as
>>>>> possible. This ResponseHeader would make sure of that.
>>>>>
>>>>> On Sat, Jun 4, 2011 at 4:04 AM, Diogo Pinheiro
>>>>> <[email protected]> wrote:
>>>>> Hi
>>>>>
>>>>> I have another question. Why the ResponseHeader has the currentVersionNo
>>>>> and currentTestsVersion ? I think we shouldn't send that information all
>>>>> the time. Just when the agent connect, and in a rpc call.
>>>>>
>>>>> What do you think ?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best,
>>>>> __
>>>>> Zubair
>>>>
>>>> ---
>>>> Adriano Monteiro Marques
>>>>
>>>> http://www.thoughtspad.com
>>>> http://www.umitproject.org
>>>> http://blog.umitproject.org
>>>> http://www.pythonbenelux.org
>>>>
>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Simplify data backup and recovery for your virtual environment with
>>>> vRanger.
>>>> Installation's a snap, and flexible recovery options mean your data is
>>>> safe,
>>>> secure and there when you need it. Discover what all the cheering's about.
>>>> Get your free trial download today.
>>>> http://p.sf.net/sfu/quest-dev2dev2
>>>> _______________________________________________
>>>> Umit-devel mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhongjie Wang
>>>> Master Candidate
>>>> Computer System Architecture
>>>> Peking University, China
>>>
>>>
>>> ---
>>> Adriano Monteiro Marques
>>>
>>> http://www.thoughtspad.com
>>> http://www.umitproject.org
>>> http://blog.umitproject.org
>>> http://www.pythonbenelux.org
>>>
>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>
>>>
>>>
>>>
>>> --
>>> Zhongjie Wang
>>> Master Candidate
>>> Computer System Architecture
>>> Peking University, China
>>
>>
>> ---
>> Adriano Monteiro Marques
>>
>> http://www.thoughtspad.com
>> http://www.umitproject.org
>> http://blog.umitproject.org
>> http://www.pythonbenelux.org
>>
>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>
>>
>>
>>
>> --
>> Best,
>> __
>> Zubair
>>
>>
>>
>> --
>> Zhongjie Wang
>> Master Candidate
>> Computer System Architecture
>> Peking University, China
>
>
> ---
> Adriano Monteiro Marques
>
> http://www.thoughtspad.com
> http://www.umitproject.org
> http://blog.umitproject.org
> http://www.pythonbenelux.org
>
> "Don't stay in bed, unless you can make money in bed." - George Burns
>
>
>
>
> --
> Zhongjie Wang
> Master Candidate
> Computer System Architecture
> Peking University, China
>
>
>
> --
> Best,
> __
> Zubair
---
Adriano Monteiro Marques
http://www.thoughtspad.com
http://www.umitproject.org
http://blog.umitproject.org
http://www.pythonbenelux.org
"Don't stay in bed, unless you can make money in bed." - George Burns
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Umit-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/umit-devel
