I've gone ahead and added testType to message Test.
Also, I've taken the liberty of adding string aggregatorPublicKey to message
registerAgentResponse. We need the aggregator's public key for agent
authentication.
Changes have been committed to the common repository.
On Thu, Jun 30, 2011 at 8:18 AM, Zhongjie Wang <[email protected]> wrote:
> Hi,
>
> On Thu, Jun 30, 2011 at 5:21 AM, Diogo Pinheiro <[email protected]
> > wrote:
>
>> Hi
>>
>> 2011/6/28 Zubair Nabi <[email protected]>
>>
>>> Guys,
>>>
>>> I propose that we add a field: required string testType
>>> to message Test to differentiate between a website test and a service
>>> test. Similar to the field in message Events.
>>>
>>
>> I agree with this
>>
>>>
>>> Also, Alan you haven't added the message that will encapsulate all other
>>> messages for p2p communication to the common proto. Can you please do so
>>> now? Or we still haven't decided that this is the best approach? Because it
>>> has my vote. Like you said, if a message doesn't exist, it will not take any
>>> space.
>>>
>>
>> This will be just for to P2P communication or for aggregator too ?
>>
>>>
>>>
> This will be also used for the aggregator sending messages to agents.
> Soon Zubair and I will figure out the p2p message format, and the
> aggregator could use them too. So don't mind that format for now.
>
>
>>
>>> On Fri, Jun 10, 2011 at 8:13 PM, Zhongjie Wang <[email protected]> wrote:
>>>
>>>> Hi Adriano,
>>>>
>>>> On Fri, Jun 10, 2011 at 11:03 PM, Adriano Monteiro Marques <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Jun 10, 2011, at 11:34 AM, Zhongjie Wang wrote:
>>>>>
>>>>> Hi Guys,
>>>>>
>>>>> On Fri, Jun 10, 2011 at 10:02 PM, Zubair Nabi <[email protected]
>>>>> > wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>>
>>>>>> If the token is sent with the message in plain text, then someone may
>>>>>>>> perform a wire-tapping attack.
>>>>>>>> And if the two party in communication has already had a handshake
>>>>>>>> progress and figured out a symmetric key
>>>>>>>> for current session, then we don't need the token i guess, the
>>>>>>>> symmetric key itself is enough for the encryption and authentication
>>>>>>>> purpose.
>>>>>>>> It's OK to keep agentID as you said.
>>>>>>>> Hm... Zubair helped me to set that there, let's just confirm with
>>>>>>>> him that we don't need it there. Your consideration is fair, but I'm
>>>>>>>> not
>>>>>>>> sure this is the only purpose the token is there.
>>>>>>>
>>>>>>> The token is also supposed to reflect the state of the agent binary.
>>>>>> For example, if the binary is updated, the symmetric key would remain the
>>>>>> same but the token would change.
>>>>>> And we aren't going to send messages in plain text. We're going to
>>>>>> encrypt each message.
>>>>>>
>>>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>>>>> fields.
>>>>>>> But since they (redirectLink, htmlResponse, htmlMedia) are
>>>>>>> "optional" in the Detail structure,
>>>>>>> they can be omitted while sending.
>>>>>>> Because when we send back Detail in Event message, we need to clone
>>>>>>> it and only send the necessary fields.
>>>>>>> Ok, sounds good. Can you get them fixed, and let the others know
>>>>>>> about your fix?
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> So that's what I propose at the beginning.
>>>>>>> All messages will be instantiate as a common message type.
>>>>>>> It has a type field and some option fields.
>>>>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>>>>> then read the type field, and choose which optional field to read.
>>>>>>> The optional fields which doesn't exists in a message don't take any
>>>>>>> spaces.
>>>>>>> e.g.
>>>>>>> message ICMMessage {
>>>>>>> required int64 agentID = 1;
>>>>>>> required int32 type = 2;
>>>>>>> optional AssignTask assignTask = 100;
>>>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>>>> optional VersionUpdate versionUpdate = 102;
>>>>>>> // extensions 100 to max;
>>>>>>> }
>>>>>>> Did you try this approach? Not sure it will work with protocol
>>>>>>> buffers.
>>>>>>
>>>>>>
>>>>>> Luis and I discussed this a while back and we came to the conclusion
>>>>>> that this won't be work. In case of p2p communication, we will have to
>>>>>> send
>>>>>> an additional message.
>>>>>> For example our send would look something like send(string type,
>>>>>> string message)
>>>>>> At the receiving end, this "type" would tell us which type of Protobuf
>>>>>> message this is.
>>>>>>
>>>>>
>>>>> Hi Zubair, I'm using the way you said now. But I think what I mentioned
>>>>> also works. I just tested that. ;)
>>>>> Here's the .proto file and test.py I used.
>>>>>
>>>>>
>>>>> Oh, now I understood your approach. We would have one type of message
>>>>> that would encapsulate all others, right?
>>>>> The problem with this approach is that the message gets to big and
>>>>> slower to process. Though it works, protobuf is got to parse the whole
>>>>> thing, and it will place several markers in the message to indicate that
>>>>> the
>>>>> field is empty. This can escalate to a few hundreds easily as we extend
>>>>> our
>>>>> app.
>>>>>
>>>>>
>>>> I was worrying about the message size. But after tested, I found even I
>>>> added more optional fields to that message. The size didn't grow, as I have
>>>> printed out. (I added ExtendedMessageType up to 10).
>>>> I think it's because the protobuf message use *unique tag number* to
>>>> indicate the field.
>>>> If an optional field doesn't exist, it will not take any space in the
>>>> message.
>>>>
>>>>
>>>>> test.proto:
>>>>>
>>>>> // this is a sample message sent from the aggregator to the agent
>>>>> message ICMMessage {
>>>>> required int32 type = 1;
>>>>>
>>>>> optional AssignTask assignTask = 100;
>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>> optional ExtendedMessageType1 em1 = 102;
>>>>> optional ExtendedMessageType2 em2 = 103;
>>>>> optional ExtendedMessageType3 em3 = 104;
>>>>> // extensions 100 to max;
>>>>> }
>>>>>
>>>>> message AssignTask {
>>>>> required int32 foo = 1;
>>>>> required int32 bar = 2;
>>>>> }
>>>>>
>>>>> message UpgradeToSuper {
>>>>> required int32 foo = 1;
>>>>> required int32 bar = 2;
>>>>> }
>>>>>
>>>>> message ExtendedMessageType1 {
>>>>> required int32 foo = 1;
>>>>> required int32 bar = 2;
>>>>> }
>>>>>
>>>>> message ExtendedMessageType2 {
>>>>> required int32 foo = 1;
>>>>> required int32 bar = 2;
>>>>> }
>>>>>
>>>>> message ExtendedMessageType3 {
>>>>> required int32 foo = 1;
>>>>> required int32 bar = 2;
>>>>> }
>>>>>
>>>>>
>>>>> test.py:
>>>>>
>>>>> import test_pb2
>>>>>
>>>>> MSG_ASSIGN_TASK = 1
>>>>> MSG_UPGRADE_TO_SUPER = 2
>>>>> MessageTypes = {1:'AssignTask', 2:'UpgradeToSuper'}
>>>>>
>>>>> msg = test_pb2.ICMMessage()
>>>>> msg.type = MSG_ASSIGN_TASK
>>>>> msg.assignTask.foo = 11
>>>>> msg.assignTask.bar = 22
>>>>> msg_send = msg.SerializeToString()
>>>>>
>>>>> print(msg)
>>>>> print(msg_send)
>>>>> print("Length: %d" % len(msg_send))
>>>>>
>>>>> msg_recv = msg_send
>>>>>
>>>>> msg1 = test_pb2.ICMMessage()
>>>>> msg1.ParseFromString(msg_recv)
>>>>> print("Got a message. Type: %s" % MessageTypes[msg1.type])
>>>>> print("Detail:")
>>>>> print(msg1.assignTask)
>>>>>
>>>>> The result:
>>>>>
>>>>> <image.png>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> On Fri, Jun 10, 2011 at 6:49 PM, Adriano Monteiro Marques <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi Alan,
>>>>>>>
>>>>>>> On Jun 10, 2011, at 12:53 AM, Zhongjie Wang wrote:
>>>>>>>
>>>>>>> Hi Adriano,
>>>>>>> Thanks for the timely response. :)
>>>>>>>
>>>>>>> On Thu, Jun 9, 2011 at 10:44 PM, Adriano Monteiro Marques <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi Alan,
>>>>>>>>
>>>>>>>> On Jun 9, 2011, at 3:01 AM, Zhongjie Wang wrote:
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> About the message format, I have some comments.
>>>>>>>> 1. Seems there're duplicate fields in SendWebsiteReport and
>>>>>>>> SendServiceReport:
>>>>>>>>
>>>>>>>> 162<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L162>
>>>>>>>>
>>>>>>>> // send_report
>>>>>>>>
>>>>>>>> 163<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L163>
>>>>>>>>
>>>>>>>> message SendWebsiteReport {
>>>>>>>>
>>>>>>>> 164<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L164>
>>>>>>>>
>>>>>>>> required RequestHeader header = 1;
>>>>>>>>
>>>>>>>> 165<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L165>
>>>>>>>>
>>>>>>>> required WebsiteReport report = 2;
>>>>>>>>
>>>>>>>> 166<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L166>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> In the RequestHeader structure to the aggregator, there're
>>>>>>>> are token and agentID. And in WebsiteReport.ICMReport, there're also
>>>>>>>> agentID. Though I didn't know the difference between token and
>>>>>>>> agentID. I
>>>>>>>> suggest we eliminate the agentID field in ICMReport.
>>>>>>>>
>>>>>>>>
>>>>>>>> Token is our to confirm to the aggregator that we are legitimate. On
>>>>>>>> the duplicated agentID, it is fine: If a node is passing a website
>>>>>>>> report
>>>>>>>> from another node, then the ICMReport will show the agentID for the
>>>>>>>> origin
>>>>>>>> node, and the RequestHeader will provide the passing agent's id.
>>>>>>>>
>>>>>>>>
>>>>>>> If the token is sent with the message in plain text, then someone may
>>>>>>> perform a wire-tapping attack.
>>>>>>> And if the two party in communication has already had a handshake
>>>>>>> progress and figured out a symmetric key
>>>>>>> for current session, then we don't need the token i guess, the
>>>>>>> symmetric key itself is enough for the encryption and authentication
>>>>>>> purpose.
>>>>>>> It's OK to keep agentID as you said.
>>>>>>>
>>>>>>>
>>>>>>> Hm... Zubair helped me to set that there, let's just confirm with him
>>>>>>> that we don't need it there. Your consideration is fair, but I'm not
>>>>>>> sure
>>>>>>> this is the only purpose the token is there.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 2. Besides that, the optional fields: redirectLink,
>>>>>>>> htmlResponse, htmlMedia, seems to be better in the WebsiteReportDetail
>>>>>>>> structure, because it's more test specific. I suppose the Report
>>>>>>>> messages
>>>>>>>> should have only two parts, the header and the detail.
>>>>>>>>
>>>>>>>>
>>>>>>>> Check the Event message... when we send back a WebsiteReportDetail
>>>>>>>> or ServiceReportDetail we don't send out those details as they're not
>>>>>>>> really
>>>>>>>> necessary.
>>>>>>>>
>>>>>>>>
>>>>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>>>>> fields.
>>>>>>> But since they (redirectLink, htmlResponse, htmlMedia) are
>>>>>>> "optional" in the Detail structure,
>>>>>>> they can be omitted while sending.
>>>>>>> Because when we send back Detail in Event message, we need to clone
>>>>>>> it and only send the necessary fields.
>>>>>>>
>>>>>>>
>>>>>>> Ok, sounds good. Can you get them fixed, and let the others know
>>>>>>> about your fix?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 3. Since we don't have a field in the message to identify
>>>>>>>> the message type, I don't know how we can create a message instance
>>>>>>>> after
>>>>>>>> receiving a bunch of raw bytes. Currently I send the message type
>>>>>>>> string
>>>>>>>> before send the message itself. But I think the better solution is tag
>>>>>>>> it in
>>>>>>>> the message structure.
>>>>>>>>
>>>>>>>>
>>>>>>>> Even if we had a field in the message, how would we retrieve it to
>>>>>>>> know the type prior to instantiating it? What we're doing on the
>>>>>>>> Aggregator
>>>>>>>> side is to presume the sort of message a given API is supposed to
>>>>>>>> send/receive. Perhaps we could do the same with the desktop agent,
>>>>>>>> what do
>>>>>>>> you think?
>>>>>>>>
>>>>>>>>
>>>>>>> So that's what I propose at the beginning.
>>>>>>> All messages will be instantiate as a common message type.
>>>>>>> It has a type field and some option fields.
>>>>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>>>>> then read the type field, and choose which optional field to read.
>>>>>>> The optional fields which doesn't exists in a message don't take any
>>>>>>> spaces.
>>>>>>> e.g.
>>>>>>> message ICMMessage {
>>>>>>> required int64 agentID = 1;
>>>>>>> required int32 type = 2;
>>>>>>>
>>>>>>> optional AssignTask assignTask = 100;
>>>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>>>> optional VersionUpdate versionUpdate = 102;
>>>>>>> // extensions 100 to max;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Did you try this approach? Not sure it will work with protocol
>>>>>>> buffers.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 4. Another slightly modification, I suggest that we use
>>>>>>>> float type for responseTime, the unit is seconds, for accuracy.
>>>>>>>>
>>>>>>>>
>>>>>>>> We're using miliseconds there as mentioned in a previous email,
>>>>>>>> that's the reason why we use integer. It saves on message size and
>>>>>>>> keeps
>>>>>>>> accuracy.
>>>>>>>>
>>>>>>>> Well noticed Alan! Keep pushing! Let me know if there still is
>>>>>>>> anything you feel like it is wrong or that is stopping you from
>>>>>>>> progressing.
>>>>>>>>
>>>>>>>>
>>>>>>> Thanks, Adriano! I'll keep on thinking and make it perfect. :)
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Cheers!
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards.
>>>>>>>> Alan
>>>>>>>>
>>>>>>>> On Sat, Jun 4, 2011 at 5:34 PM, Adriano Monteiro Marques <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Right, mainly for nodes who stays connected for long periods of
>>>>>>>>> time. Also, if we want a test to be run ASAP, we need to send this
>>>>>>>>> info
>>>>>>>>> everytime so the node knows of a brand new test version, updates
>>>>>>>>> their test
>>>>>>>>> set and run them ASAP. On the software version, if we figure any
>>>>>>>>> security
>>>>>>>>> issue, the nodes will also update ASAP.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jun 4, 2011, at 6:22 AM, Zubair Nabi wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I think the idea was to keep the binary and test versions as fresh
>>>>>>>>> as possible. This ResponseHeader would make sure of that.
>>>>>>>>>
>>>>>>>>> On Sat, Jun 4, 2011 at 4:04 AM, Diogo Pinheiro <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> I have another question. Why the ResponseHeader has the
>>>>>>>>>> currentVersionNo and currentTestsVersion ? I think we shouldn't send
>>>>>>>>>> that
>>>>>>>>>> information all the time. Just when the agent connect, and in a rpc
>>>>>>>>>> call.
>>>>>>>>>>
>>>>>>>>>> What do you think ?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best,
>>>>>>>>> __
>>>>>>>>> Zubair
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Adriano Monteiro Marques
>>>>>>>>>
>>>>>>>>> http://www.thoughtspad.com
>>>>>>>>> http://www.umitproject.org
>>>>>>>>> http://blog.umitproject.org
>>>>>>>>> http://www.pythonbenelux.org
>>>>>>>>>
>>>>>>>>> "Don't stay in bed, unless you can make money in bed." - George
>>>>>>>>> Burns
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>> Simplify data backup and recovery for your virtual environment with
>>>>>>>>> vRanger.
>>>>>>>>> Installation's a snap, and flexible recovery options mean your data
>>>>>>>>> is safe,
>>>>>>>>> secure and there when you need it. Discover what all the cheering's
>>>>>>>>> about.
>>>>>>>>> Get your free trial download today.
>>>>>>>>> http://p.sf.net/sfu/quest-dev2dev2
>>>>>>>>> _______________________________________________
>>>>>>>>> Umit-devel mailing list
>>>>>>>>> [email protected]
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Zhongjie Wang
>>>>>>>> Master Candidate
>>>>>>>> Computer System Architecture
>>>>>>>> Peking University, China
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Adriano Monteiro Marques
>>>>>>>>
>>>>>>>> http://www.thoughtspad.com
>>>>>>>> http://www.umitproject.org
>>>>>>>> http://blog.umitproject.org
>>>>>>>> http://www.pythonbenelux.org
>>>>>>>>
>>>>>>>> "Don't stay in bed, unless you can make money in bed." - George
>>>>>>>> Burns
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Zhongjie Wang
>>>>>>> Master Candidate
>>>>>>> Computer System Architecture
>>>>>>> Peking University, China
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Adriano Monteiro Marques
>>>>>>>
>>>>>>> http://www.thoughtspad.com
>>>>>>> http://www.umitproject.org
>>>>>>> http://blog.umitproject.org
>>>>>>> http://www.pythonbenelux.org
>>>>>>>
>>>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best,
>>>>>> __
>>>>>> Zubair
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhongjie Wang
>>>>> Master Candidate
>>>>> Computer System Architecture
>>>>> Peking University, China
>>>>>
>>>>>
>>>>> ---
>>>>> Adriano Monteiro Marques
>>>>>
>>>>> http://www.thoughtspad.com
>>>>> http://www.umitproject.org
>>>>> http://blog.umitproject.org
>>>>> http://www.pythonbenelux.org
>>>>>
>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Zhongjie Wang
>>>> Master Candidate
>>>> Computer System Architecture
>>>> Peking University, China
>>>>
>>>
>>>
>>>
>>> --
>>> Best,
>>> __
>>> Zubair
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> All of the data generated in your IT infrastructure is seriously
>>> valuable.
>>> Why? It contains a definitive record of application performance, security
>>> threats, fraudulent activity, and more. Splunk takes this data and makes
>>> sense of it. IT sense. And common sense.
>>> http://p.sf.net/sfu/splunk-d2d-c2
>>>
>>> _______________________________________________
>>> Umit-devel mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>>
>>>
>>
>>
>> --
>> Cumprimentos
>> Diogo Pinheiro
>>
>>
>
>
> --
> Zhongjie Wang
> Master Candidate
> Computer System Architecture
> Peking University, China
>
--
Best,
__
Zubair
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Umit-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/umit-devel
