Hi
2011/6/28 Zubair Nabi <[email protected]>
> Guys,
>
> I propose that we add a field: required string testType
> to message Test to differentiate between a website test and a service test.
> Similar to the field in message Events.
>
I agree with this
>
> Also, Alan you haven't added the message that will encapsulate all other
> messages for p2p communication to the common proto. Can you please do so
> now? Or we still haven't decided that this is the best approach? Because it
> has my vote. Like you said, if a message doesn't exist, it will not take any
> space.
>
This will be just for to P2P communication or for aggregator too ?
>
>
> On Fri, Jun 10, 2011 at 8:13 PM, Zhongjie Wang <[email protected]> wrote:
>
>> Hi Adriano,
>>
>> On Fri, Jun 10, 2011 at 11:03 PM, Adriano Monteiro Marques <
>> [email protected]> wrote:
>>
>>> Hi,
>>>
>>> On Jun 10, 2011, at 11:34 AM, Zhongjie Wang wrote:
>>>
>>> Hi Guys,
>>>
>>> On Fri, Jun 10, 2011 at 10:02 PM, Zubair Nabi
>>> <[email protected]>wrote:
>>>
>>>> Hi Guys,
>>>>
>>>>
>>>> If the token is sent with the message in plain text, then someone may
>>>>>> perform a wire-tapping attack.
>>>>>> And if the two party in communication has already had a handshake
>>>>>> progress and figured out a symmetric key
>>>>>> for current session, then we don't need the token i guess, the
>>>>>> symmetric key itself is enough for the encryption and authentication
>>>>>> purpose.
>>>>>> It's OK to keep agentID as you said.
>>>>>> Hm... Zubair helped me to set that there, let's just confirm with him
>>>>>> that we don't need it there. Your consideration is fair, but I'm not sure
>>>>>> this is the only purpose the token is there.
>>>>>
>>>>> The token is also supposed to reflect the state of the agent binary.
>>>> For example, if the binary is updated, the symmetric key would remain the
>>>> same but the token would change.
>>>> And we aren't going to send messages in plain text. We're going to
>>>> encrypt each message.
>>>>
>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>>> fields.
>>>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional"
>>>>> in the Detail structure,
>>>>> they can be omitted while sending.
>>>>> Because when we send back Detail in Event message, we need to clone it
>>>>> and only send the necessary fields.
>>>>> Ok, sounds good. Can you get them fixed, and let the others know about
>>>>> your fix?
>>>>
>>>> +1
>>>>
>>>> So that's what I propose at the beginning.
>>>>> All messages will be instantiate as a common message type.
>>>>> It has a type field and some option fields.
>>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>>> then read the type field, and choose which optional field to read.
>>>>> The optional fields which doesn't exists in a message don't take any
>>>>> spaces.
>>>>> e.g.
>>>>> message ICMMessage {
>>>>> required int64 agentID = 1;
>>>>> required int32 type = 2;
>>>>> optional AssignTask assignTask = 100;
>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>> optional VersionUpdate versionUpdate = 102;
>>>>> // extensions 100 to max;
>>>>> }
>>>>> Did you try this approach? Not sure it will work with protocol buffers.
>>>>
>>>>
>>>> Luis and I discussed this a while back and we came to the conclusion
>>>> that this won't be work. In case of p2p communication, we will have to send
>>>> an additional message.
>>>> For example our send would look something like send(string type, string
>>>> message)
>>>> At the receiving end, this "type" would tell us which type of Protobuf
>>>> message this is.
>>>>
>>>
>>> Hi Zubair, I'm using the way you said now. But I think what I mentioned
>>> also works. I just tested that. ;)
>>> Here's the .proto file and test.py I used.
>>>
>>>
>>> Oh, now I understood your approach. We would have one type of message
>>> that would encapsulate all others, right?
>>> The problem with this approach is that the message gets to big and slower
>>> to process. Though it works, protobuf is got to parse the whole thing, and
>>> it will place several markers in the message to indicate that the field is
>>> empty. This can escalate to a few hundreds easily as we extend our app.
>>>
>>>
>> I was worrying about the message size. But after tested, I found even I
>> added more optional fields to that message. The size didn't grow, as I have
>> printed out. (I added ExtendedMessageType up to 10).
>> I think it's because the protobuf message use *unique tag number* to
>> indicate the field.
>> If an optional field doesn't exist, it will not take any space in the
>> message.
>>
>>
>>> test.proto:
>>>
>>> // this is a sample message sent from the aggregator to the agent
>>> message ICMMessage {
>>> required int32 type = 1;
>>>
>>> optional AssignTask assignTask = 100;
>>> optional UpgradeToSuper upgradeToSuper = 101;
>>> optional ExtendedMessageType1 em1 = 102;
>>> optional ExtendedMessageType2 em2 = 103;
>>> optional ExtendedMessageType3 em3 = 104;
>>> // extensions 100 to max;
>>> }
>>>
>>> message AssignTask {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message UpgradeToSuper {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message ExtendedMessageType1 {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message ExtendedMessageType2 {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>> message ExtendedMessageType3 {
>>> required int32 foo = 1;
>>> required int32 bar = 2;
>>> }
>>>
>>>
>>> test.py:
>>>
>>> import test_pb2
>>>
>>> MSG_ASSIGN_TASK = 1
>>> MSG_UPGRADE_TO_SUPER = 2
>>> MessageTypes = {1:'AssignTask', 2:'UpgradeToSuper'}
>>>
>>> msg = test_pb2.ICMMessage()
>>> msg.type = MSG_ASSIGN_TASK
>>> msg.assignTask.foo = 11
>>> msg.assignTask.bar = 22
>>> msg_send = msg.SerializeToString()
>>>
>>> print(msg)
>>> print(msg_send)
>>> print("Length: %d" % len(msg_send))
>>>
>>> msg_recv = msg_send
>>>
>>> msg1 = test_pb2.ICMMessage()
>>> msg1.ParseFromString(msg_recv)
>>> print("Got a message. Type: %s" % MessageTypes[msg1.type])
>>> print("Detail:")
>>> print(msg1.assignTask)
>>>
>>> The result:
>>>
>>> <image.png>
>>>
>>>
>>>
>>>>
>>>> On Fri, Jun 10, 2011 at 6:49 PM, Adriano Monteiro Marques <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Alan,
>>>>>
>>>>> On Jun 10, 2011, at 12:53 AM, Zhongjie Wang wrote:
>>>>>
>>>>> Hi Adriano,
>>>>> Thanks for the timely response. :)
>>>>>
>>>>> On Thu, Jun 9, 2011 at 10:44 PM, Adriano Monteiro Marques <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hi Alan,
>>>>>>
>>>>>> On Jun 9, 2011, at 3:01 AM, Zhongjie Wang wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> About the message format, I have some comments.
>>>>>> 1. Seems there're duplicate fields in SendWebsiteReport and
>>>>>> SendServiceReport:
>>>>>>
>>>>>> 162<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L162>
>>>>>>
>>>>>> // send_report
>>>>>>
>>>>>> 163<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L163>
>>>>>>
>>>>>> message SendWebsiteReport {
>>>>>>
>>>>>> 164<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L164>
>>>>>>
>>>>>> required RequestHeader header = 1;
>>>>>>
>>>>>> 165<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L165>
>>>>>>
>>>>>> required WebsiteReport report = 2;
>>>>>>
>>>>>> 166<http://dev.umitproject.org/projects/icm/repository/revisions/master/entry/proto/messages.proto#L166>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> In the RequestHeader structure to the aggregator, there're are
>>>>>> token and agentID. And in WebsiteReport.ICMReport, there're also agentID.
>>>>>> Though I didn't know the difference between token and agentID. I suggest
>>>>>> we
>>>>>> eliminate the agentID field in ICMReport.
>>>>>>
>>>>>>
>>>>>> Token is our to confirm to the aggregator that we are legitimate. On
>>>>>> the duplicated agentID, it is fine: If a node is passing a website report
>>>>>> from another node, then the ICMReport will show the agentID for the
>>>>>> origin
>>>>>> node, and the RequestHeader will provide the passing agent's id.
>>>>>>
>>>>>>
>>>>> If the token is sent with the message in plain text, then someone may
>>>>> perform a wire-tapping attack.
>>>>> And if the two party in communication has already had a handshake
>>>>> progress and figured out a symmetric key
>>>>> for current session, then we don't need the token i guess, the
>>>>> symmetric key itself is enough for the encryption and authentication
>>>>> purpose.
>>>>> It's OK to keep agentID as you said.
>>>>>
>>>>>
>>>>> Hm... Zubair helped me to set that there, let's just confirm with him
>>>>> that we don't need it there. Your consideration is fair, but I'm not sure
>>>>> this is the only purpose the token is there.
>>>>>
>>>>>
>>>>>
>>>>>> 2. Besides that, the optional fields: redirectLink,
>>>>>> htmlResponse, htmlMedia, seems to be better in the WebsiteReportDetail
>>>>>> structure, because it's more test specific. I suppose the Report messages
>>>>>> should have only two parts, the header and the detail.
>>>>>>
>>>>>>
>>>>>> Check the Event message... when we send back a WebsiteReportDetail or
>>>>>> ServiceReportDetail we don't send out those details as they're not really
>>>>>> necessary.
>>>>>>
>>>>>>
>>>>> Yes, in the Event message we don't need to send back the unnecessary
>>>>> fields.
>>>>> But since they (redirectLink, htmlResponse, htmlMedia) are "optional"
>>>>> in the Detail structure,
>>>>> they can be omitted while sending.
>>>>> Because when we send back Detail in Event message, we need to clone it
>>>>> and only send the necessary fields.
>>>>>
>>>>>
>>>>> Ok, sounds good. Can you get them fixed, and let the others know about
>>>>> your fix?
>>>>>
>>>>>
>>>>>
>>>>>> 3. Since we don't have a field in the message to identify the
>>>>>> message type, I don't know how we can create a message instance after
>>>>>> receiving a bunch of raw bytes. Currently I send the message type string
>>>>>> before send the message itself. But I think the better solution is tag
>>>>>> it in
>>>>>> the message structure.
>>>>>>
>>>>>>
>>>>>> Even if we had a field in the message, how would we retrieve it to
>>>>>> know the type prior to instantiating it? What we're doing on the
>>>>>> Aggregator
>>>>>> side is to presume the sort of message a given API is supposed to
>>>>>> send/receive. Perhaps we could do the same with the desktop agent, what
>>>>>> do
>>>>>> you think?
>>>>>>
>>>>>>
>>>>> So that's what I propose at the beginning.
>>>>> All messages will be instantiate as a common message type.
>>>>> It has a type field and some option fields.
>>>>> The receiver will instantiate it as a common ICMMessage for example,
>>>>> then read the type field, and choose which optional field to read.
>>>>> The optional fields which doesn't exists in a message don't take any
>>>>> spaces.
>>>>> e.g.
>>>>> message ICMMessage {
>>>>> required int64 agentID = 1;
>>>>> required int32 type = 2;
>>>>>
>>>>> optional AssignTask assignTask = 100;
>>>>> optional UpgradeToSuper upgradeToSuper = 101;
>>>>> optional VersionUpdate versionUpdate = 102;
>>>>> // extensions 100 to max;
>>>>> }
>>>>>
>>>>>
>>>>> Did you try this approach? Not sure it will work with protocol buffers.
>>>>>
>>>>>
>>>>>
>>>>>> 4. Another slightly modification, I suggest that we use float
>>>>>> type for responseTime, the unit is seconds, for accuracy.
>>>>>>
>>>>>>
>>>>>> We're using miliseconds there as mentioned in a previous email, that's
>>>>>> the reason why we use integer. It saves on message size and keeps
>>>>>> accuracy.
>>>>>>
>>>>>> Well noticed Alan! Keep pushing! Let me know if there still is
>>>>>> anything you feel like it is wrong or that is stopping you from
>>>>>> progressing.
>>>>>>
>>>>>>
>>>>> Thanks, Adriano! I'll keep on thinking and make it perfect. :)
>>>>>
>>>>>
>>>>>>
>>>>>> Cheers!
>>>>>>
>>>>>>
>>>>>> Best Regards.
>>>>>> Alan
>>>>>>
>>>>>> On Sat, Jun 4, 2011 at 5:34 PM, Adriano Monteiro Marques <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Right, mainly for nodes who stays connected for long periods of time.
>>>>>>> Also, if we want a test to be run ASAP, we need to send this info
>>>>>>> everytime
>>>>>>> so the node knows of a brand new test version, updates their test set
>>>>>>> and
>>>>>>> run them ASAP. On the software version, if we figure any security
>>>>>>> issue, the
>>>>>>> nodes will also update ASAP.
>>>>>>>
>>>>>>>
>>>>>>> On Jun 4, 2011, at 6:22 AM, Zubair Nabi wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I think the idea was to keep the binary and test versions as fresh as
>>>>>>> possible. This ResponseHeader would make sure of that.
>>>>>>>
>>>>>>> On Sat, Jun 4, 2011 at 4:04 AM, Diogo Pinheiro <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I have another question. Why the ResponseHeader has the
>>>>>>>> currentVersionNo and currentTestsVersion ? I think we shouldn't send
>>>>>>>> that
>>>>>>>> information all the time. Just when the agent connect, and in a rpc
>>>>>>>> call.
>>>>>>>>
>>>>>>>> What do you think ?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best,
>>>>>>> __
>>>>>>> Zubair
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Adriano Monteiro Marques
>>>>>>>
>>>>>>> http://www.thoughtspad.com
>>>>>>> http://www.umitproject.org
>>>>>>> http://blog.umitproject.org
>>>>>>> http://www.pythonbenelux.org
>>>>>>>
>>>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------------
>>>>>>> Simplify data backup and recovery for your virtual environment with
>>>>>>> vRanger.
>>>>>>> Installation's a snap, and flexible recovery options mean your data
>>>>>>> is safe,
>>>>>>> secure and there when you need it. Discover what all the cheering's
>>>>>>> about.
>>>>>>> Get your free trial download today.
>>>>>>> http://p.sf.net/sfu/quest-dev2dev2
>>>>>>> _______________________________________________
>>>>>>> Umit-devel mailing list
>>>>>>> [email protected]
>>>>>>> https://lists.sourceforge.net/lists/listinfo/umit-devel
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Zhongjie Wang
>>>>>> Master Candidate
>>>>>> Computer System Architecture
>>>>>> Peking University, China
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Adriano Monteiro Marques
>>>>>>
>>>>>> http://www.thoughtspad.com
>>>>>> http://www.umitproject.org
>>>>>> http://blog.umitproject.org
>>>>>> http://www.pythonbenelux.org
>>>>>>
>>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Zhongjie Wang
>>>>> Master Candidate
>>>>> Computer System Architecture
>>>>> Peking University, China
>>>>>
>>>>>
>>>>> ---
>>>>> Adriano Monteiro Marques
>>>>>
>>>>> http://www.thoughtspad.com
>>>>> http://www.umitproject.org
>>>>> http://blog.umitproject.org
>>>>> http://www.pythonbenelux.org
>>>>>
>>>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best,
>>>> __
>>>> Zubair
>>>>
>>>
>>>
>>>
>>> --
>>> Zhongjie Wang
>>> Master Candidate
>>> Computer System Architecture
>>> Peking University, China
>>>
>>>
>>> ---
>>> Adriano Monteiro Marques
>>>
>>> http://www.thoughtspad.com
>>> http://www.umitproject.org
>>> http://blog.umitproject.org
>>> http://www.pythonbenelux.org
>>>
>>> "Don't stay in bed, unless you can make money in bed." - George Burns
>>>
>>>
>>
>>
>> --
>> Zhongjie Wang
>> Master Candidate
>> Computer System Architecture
>> Peking University, China
>>
>
>
>
> --
> Best,
> __
> Zubair
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> Umit-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/umit-devel
>
>
--
Cumprimentos
Diogo Pinheiro
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Umit-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/umit-devel
