Hi,
On Thu, Jun 30, 2011 at 5:21 AM, Diogo Pinheiro
<[email protected]>wrote:
> Hi
>
> 2011/6/28 Zubair Nabi <[email protected]>
>
>> Guys,
>>
>> I propose that we add a field: required string testType
>> to message Test to differentiate between a website test and a service
>> test. Similar to the field in message Events.
>>
>
> I agree with this
>
>>
>> Also, Alan you haven't added the message that will encapsulate all other
>> messages for p2p communication to the common proto. Can you please do so
>> now? Or we still haven't decided that this is the best approach? Because it
>> has my vote. Like you said, if a message doesn't exist, it will not take any
>> space.
>>
>
> This will be just for to P2P communication or for aggregator too ?
>
>>
>>
This will be also used for the aggregator sending messages to agents.
Soon Zubair and I will figure out the p2p message format, and the aggregator
could use them too. So don't mind that format for now.
>
>> On Fri, Jun 10, 2011 at 8:13 PM, Zhongjie Wang <[email protected]> wrote:
>>
>>> Hi Adriano,
>>>
>>> On Fri, Jun 10, 2011 at 11:03 PM, Adriano Monteiro Marques <
>>> [email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Jun 10, 2011, at 11:34 AM, Zhongjie Wang wrote:
>>>>
>>>> Hi Guys,
>>>>
>>>> On Fri, Jun 10, 2011 at 10:02 PM, Zubair Nabi
>>>> <[email protected]>wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>>
>>>>> If the token is sent with the message in plain text, then someone may
>>>>>>> perform a wire-tapping attack.
>>>>>>> And if the two party in communication has already had a handshake
>>>>>>> progress and figured out a symmetric key
>>>>>>> for current session, then we don't need the token i guess, the
>>>>>>> symmetric key itself is enough for the encryption and authentication
>>>>>>> purpose.
>>>>>>> It's OK to keep agentID as you said.
>>>>>>> Hm... Zubair helped me to set that there, let's just confirm with him
>>>>>>> that we don't need it there. Your consideration is fair, but I'm not
>>>>>>> sure
>>>>>>> this is the only purpose the token is there.
>>>>>>
>>>>>> The token is also supposed to reflect the state of the agent binary.
>>>>> For example, if the binary is updated, the symmetric key would remain the
>>>>> same but the token would change.
>>>>> And we aren't going to send messages in plain text. We're going to
>>>>> encrypt each message.
>>>>>
>>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>>>> fields.
>>>>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional"
>>>>>> in the Detail structure,
>>>>>> they can be omitted while sending.
>>>>>> Because when we send back Detail in Event message, we need to clone it
>>>>>> and only send the necessary fields.
>>>>>> Ok, sounds good. Can you get them fixed, and let the others know about
>>>>>> your fix?
>>>>>
>>>>> +1
>>>>>
>>>>> So that's what I propose at the beginning.
>>>>>> All messages will be instantiate as a common message type.
>>>>>> It has a type field and some option fields.
>>>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>>>> then read the type field, and choose which optional field to read.
>>>>>> The optional fields which doesn't exists in a message don't take any
>>>>>> spaces.
>>>>>> e.g.
>>>>>> message ICMMessage {
>>>>>> required int64 agentID = 1;
>>>>>> required int32 type = 2;
>>>>>> optional AssignTask assignTask = 100;
>>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>>> optional VersionUpdate versionUpdate = 102;
>>>>>> // extensions 100 to max;
>>>>>> }
>>>>>> Did you try this approach? Not sure it will work with protocol
>>>>>> buffers.
>>>>>
>>>>>
>>>>> Luis and I discussed this a while back and we came to the conclusion
>>>>> that this won't be work. In case of p2p communication, we will have to
>>>>> send
>>>>> an additional message.
>>>>> For example our send would look something like send(string type, string
>>>>> message)
>>>>> At the receiving end, this "type" would tell us which type of Protobuf
>>>>> message this is.
>>>>>
>>>>
>>>> Hi Zubair, I'm using the way you said now. But I think what I mentioned
>>>> also works. I just tested that. ;)
>>>> Here's the .proto file and test.py I used.
>>>>
>>>>
>>>> Oh, now I understood your approach. We would have one type of message
>>>> that would encapsulate all others, right?
>>>> The problem with this approach is that the message gets to big and
>>>> slower to process. Though it works, protobuf is got to parse the whole
>>>> thing, and it will place several markers in the message to indicate that
>>>> the
>>>> field is empty. This can escalate to a few hundreds easily as we extend our
>>>> app.
>>>>
>>>>
>>> I was worrying about the message size. But after tested, I found even I
>>> added more optional fields to that message. The size didn't grow, as I have
>>> printed out. (I added ExtendedMessageType up to 10).
>>> I think it's because the protobuf message use *unique tag number* to
>>> indicate the field.
>>> If an optional field doesn't exist, it will not take any space in the
>>> message.
>>>
>>>
>>>> test.proto:
>>>>
>>>> // this is a sample message sent from the aggregator to the agent
>>>> message ICMMessage {
>>>> required int32 type = 1;
>>>>
>>>> optional AssignTask assignTask = 100;
>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>> optional ExtendedMessageType1 em1 = 102;
>>>> optional ExtendedMessageType2 em2 = 103;
>>>> optional ExtendedMessageType3 em3 = 104;
>>>> // extensions 100 to max;
>>>> }
>>>>
>>>> message AssignTask {
>>>> required int32 foo = 1;
>>>> required int32 bar = 2;
>>>> }
>>>>
>>>> message UpgradeToSuper {
>>>> required int32 foo = 1;
>>>> required int32 bar = 2;
>>>> }
>>>>
>>>> message ExtendedMessageType1 {
>>>> required int32 foo = 1;
>>>> required int32 bar = 2;
>>>> }
>>>>
>>>> message ExtendedMessageType2 {
>>>> required int32 foo = 1;
>>>> required int32 bar = 2;
>>>> }
>>>>
>>>> message ExtendedMessageType3 {
>>>> required int32 foo = 1;
>>>> required int32 bar = 2;
>>>> }
>>>>
>>>>
>>>> test.py:
>>>>
>>>> import test_pb2
>>>>
>>>> MSG_ASSIGN_TASK = 1
>>>> MSG_UPGRADE_TO_SUPER = 2
>>>> MessageTypes = {1:'AssignTask', 2:'UpgradeToSuper'}
>>>>
>>>> msg = test_pb2.ICMMessage()
>>>> msg.type = MSG_ASSIGN_TASK
>>>> msg.assignTask.foo = 11
>>>> msg.assignTask.bar = 22
>>>> msg_send = msg.SerializeToString()
>>>>
>>>> print(msg)
>>>> print(msg_send)
>>>> print("Length: %d" % len(msg_send))
>>>>
>>>> msg_recv = msg_send
>>>>
>>>> msg1 = test_pb2.ICMMessage()
>>>> msg1.ParseFromString(msg_recv)
>>>> print("Got a message. Type: %s" % MessageTypes[msg1.type])
>>>> print("Detail:")
>>>> print(msg1.assignTask)
>>>>
>>>> The result:
>>>>
>>>> <image.png>
>>>>
>>>>
>>>>
>>>>>
>>>>> On Fri, Jun 10, 2011 at 6:49 PM, Adriano Monteiro Marques <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Alan,
>>>>>>
>>>>>> On Jun 10, 2011, at 12:53 AM, Zhongjie Wang wrote:
>>>>>>
>>>>>> Hi Adriano,
>>>>>> Thanks for the timely response. :)
>>>>>>
>>>>>> On Thu, Jun 9, 2011 at 10:44 PM, Adriano Monteiro Marques <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi Alan,
>>>>>>>
>>>>>>> On Jun 9, 2011, at 3:01 AM, Zhongjie Wang wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> About the message format, I have some comments.
>>>>>>> 1. Seems there're duplicate fields in SendWebsiteReport and
>>>>>>> SendServiceReport:
>>>>>>>
>>>>>>> 162<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L162>
>>>>>>>
>>>>>>> // send_report
>>>>>>>
>>>>>>> 163<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L163>
>>>>>>>
>>>>>>> message SendWebsiteReport {
>>>>>>>
>>>>>>> 164<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L164>
>>>>>>>
>>>>>>> required RequestHeader header = 1;
>>>>>>>
>>>>>>> 165<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L165>
>>>>>>>
>>>>>>> required WebsiteReport report = 2;
>>>>>>>
>>>>>>> 166<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L166>
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> In the RequestHeader structure to the aggregator, there're are
>>>>>>> token and agentID. And in WebsiteReport.ICMReport, there're also
>>>>>>> agentID.
>>>>>>> Though I didn't know the difference between token and agentID. I
>>>>>>> suggest we
>>>>>>> eliminate the agentID field in ICMReport.
>>>>>>>
>>>>>>>
>>>>>>> Token is our to confirm to the aggregator that we are legitimate. On
>>>>>>> the duplicated agentID, it is fine: If a node is passing a website
>>>>>>> report
>>>>>>> from another node, then the ICMReport will show the agentID for the
>>>>>>> origin
>>>>>>> node, and the RequestHeader will provide the passing agent's id.
>>>>>>>
>>>>>>>
>>>>>> If the token is sent with the message in plain text, then someone may
>>>>>> perform a wire-tapping attack.
>>>>>> And if the two party in communication has already had a handshake
>>>>>> progress and figured out a symmetric key
>>>>>> for current session, then we don't need the token i guess, the
>>>>>> symmetric key itself is enough for the encryption and authentication
>>>>>> purpose.
>>>>>> It's OK to keep agentID as you said.
>>>>>>
>>>>>>
>>>>>> Hm... Zubair helped me to set that there, let's just confirm with him
>>>>>> that we don't need it there. Your consideration is fair, but I'm not sure
>>>>>> this is the only purpose the token is there.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 2. Besides that, the optional fields: redirectLink,
>>>>>>> htmlResponse, htmlMedia, seems to be better in the WebsiteReportDetail
>>>>>>> structure, because it's more test specific. I suppose the Report
>>>>>>> messages
>>>>>>> should have only two parts, the header and the detail.
>>>>>>>
>>>>>>>
>>>>>>> Check the Event message... when we send back a WebsiteReportDetail or
>>>>>>> ServiceReportDetail we don't send out those details as they're not
>>>>>>> really
>>>>>>> necessary.
>>>>>>>
>>>>>>>
>>>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>>>> fields.
>>>>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional"
>>>>>> in the Detail structure,
>>>>>> they can be omitted while sending.
>>>>>> Because when we send back Detail in Event message, we need to clone it
>>>>>> and only send the necessary fields.
>>>>>>
>>>>>>
>>>>>> Ok, sounds good. Can you get them fixed, and let the others know about
>>>>>> your fix?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 3. Since we don't have a field in the message to identify
>>>>>>> the message type, I don't know how we can create a message instance
>>>>>>> after
>>>>>>> receiving a bunch of raw bytes. Currently I send the message type string
>>>>>>> before send the message itself. But I think the better solution is tag
>>>>>>> it in
>>>>>>> the message structure.
>>>>>>>
>>>>>>>
>>>>>>> Even if we had a field in the message, how would we retrieve it to
>>>>>>> know the type prior to instantiating it? What we're doing on the
>>>>>>> Aggregator
>>>>>>> side is to presume the sort of message a given API is supposed to
>>>>>>> send/receive. Perhaps we could do the same with the desktop agent, what
>>>>>>> do
>>>>>>> you think?
>>>>>>>
>>>>>>>
>>>>>> So that's what I propose at the beginning.
>>>>>> All messages will be instantiate as a common message type.
>>>>>> It has a type field and some option fields.
>>>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>>>> then read the type field, and choose which optional field to read.
>>>>>> The optional fields which doesn't exists in a message don't take any
>>>>>> spaces.
>>>>>> e.g.
>>>>>> message ICMMessage {
>>>>>> required int64 agentID = 1;
>>>>>> required int32 type = 2;
>>>>>>
>>>>>> optional AssignTask assignTask = 100;
>>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>>> optional VersionUpdate versionUpdate = 102;
>>>>>> // extensions 100 to max;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Did you try this approach? Not sure it will work with protocol
>>>>>> buffers.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 4. Another slightly modification, I suggest that we use
>>>>>>> float type for responseTime, the unit is seconds, for accuracy.
>>>>>>>
>>>>>>>
>>>>>>> We're using miliseconds there as mentioned in a previous email,
>>>>>>> that's the reason why we use integer. It saves on message size and keeps
>>>>>>> accuracy.
>>>>>>>
>>>>>>> Well noticed Alan! Keep pushing! Let me know if there still is
>>>>>>> anything you feel like it is wrong or that is stopping you from
>>>>>>> progressing.
>>>>>>>
>>>>>>>
>>>>>> Thanks, Adriano! I'll keep on thinking and make it perfect. :)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Cheers!
>>>>>>>
>>>>>>>
>>>>>>> Best Regards.
>>>>>>> Alan
>>>>>>>
>>>>>>> On Sat, Jun 4, 2011 at 5:34 PM, Adriano Monteiro Marques <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Right, mainly for nodes who stays connected for long periods of
>>>>>>>> time. Also, if we want a test to be run ASAP, we need to send this info
>>>>>>>> everytime so the node knows of a brand new test version, updates their
>>>>>>>> test
>>>>>>>> set and run them ASAP. On the software version, if we figure any
>>>>>>>> security
>>>>>>>> issue, the nodes will also update ASAP.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jun 4, 2011, at 6:22 AM, Zubair Nabi wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I think the idea was to keep the binary and test versions as fresh
>>>>>>>> as possible. This ResponseHeader would make sure of that.
>>>>>>>>
>>>>>>>> On Sat, Jun 4, 2011 at 4:04 AM, Diogo Pinheiro <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I have another question. Why the ResponseHeader has the
>>>>>>>>> currentVersionNo and currentTestsVersion ? I think we shouldn't send
>>>>>>>>> that
>>>>>>>>> information all the time. Just when the agent connect, and in a rpc
>>>>>>>>> call.
>>>>>>>>>
>>>>>>>>> What do you think ?
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best,
>>>>>>>> __
>>>>>>>> Zubair
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Adriano Monteiro Marques
>>>>>>>>
>>>>>>>> http://www.thoughtspad.com
>>>>>>>> http://www.umitproject.org
>>>>>>>> http://blog.umitproject.org
>>>>>>>> http://www.pythonbenelux.org
>>>>>>>>
>>>>>>>> "Don't stay in bed, unless you can make money in bed." - George
>>>>>>>> Burns
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>> Simplify data backup and recovery for your virtual environment with
>>>>>>>> vRanger.
>>>>>>>> Installation's a snap, and flexible recovery options mean your data
>>>>>>>> is safe,
>>>>>>>> secure and there when you need it. Discover what all the cheering's
>>>>>>>> about.
>>>>>>>> Get your free trial download today.
>>>>>>>> http://p.sf.net/sfu/quest-dev2dev2
>>>>>>>> _______________________________________________
>>>>>>>> Umit-devel mailing list
>>>>>>>> [email protected]
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Zhongjie Wang
>>>>>>> Master Candidate
>>>>>>> Computer System Architecture
>>>>>>> Peking University, China
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Adriano Monteiro Marques
>>>>>>>
>>>>>>> http://www.thoughtspad.com
>>>>>>> http://www.umitproject.org
>>>>>>> http://blog.umitproject.org
>>>>>>> http://www.pythonbenelux.org
>>>>>>>
>>>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Zhongjie Wang
>>>>>> Master Candidate
>>>>>> Computer System Architecture
>>>>>> Peking University, China
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Adriano Monteiro Marques
>>>>>>
>>>>>> http://www.thoughtspad.com
>>>>>> http://www.umitproject.org
>>>>>> http://blog.umitproject.org
>>>>>> http://www.pythonbenelux.org
>>>>>>
>>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best,
>>>>> __
>>>>> Zubair
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Zhongjie Wang
>>>> Master Candidate
>>>> Computer System Architecture
>>>> Peking University, China
>>>>
>>>>
>>>> ---
>>>> Adriano Monteiro Marques
>>>>
>>>> http://www.thoughtspad.com
>>>> http://www.umitproject.org
>>>> http://blog.umitproject.org
>>>> http://www.pythonbenelux.org
>>>>
>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>
>>>>
>>>
>>>
>>> --
>>> Zhongjie Wang
>>> Master Candidate
>>> Computer System Architecture
>>> Peking University, China
>>>
>>
>>
>>
>> --
>> Best,
>> __
>> Zubair
>>
>>
>> ------------------------------------------------------------------------------
>> All of the data generated in your IT infrastructure is seriously valuable.
>> Why? It contains a definitive record of application performance, security
>> threats, fraudulent activity, and more. Splunk takes this data and makes
>> sense of it. IT sense. And common sense.
>> http://p.sf.net/sfu/splunk-d2d-c2
>>
>> _______________________________________________
>> Umit-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>
>>
>
>
> --
> Cumprimentos
> Diogo Pinheiro
>
>
--
Zhongjie Wang
Master Candidate
Computer System Architecture
Peking University, China
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Umit-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/umit-devel
