Re: [Wireshark-dev] Fields offsets & tree hierarchy questions

2017-07-14 Thread Sultan, Hassan via Wireshark-dev 


> -Original Message-
> From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf
> Of Jeff Morriss
> Sent: Friday, July 14, 2017 11:32 AM
> To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
> Subject: Re: [Wireshark-dev] Fields offsets & tree hierarchy questions
> 
> 
> 
> On Fri, Jul 14, 2017 at 2:01 PM, Sultan, Hassan via Wireshark-dev  d...@wireshark.org <mailto:wireshark-dev@wireshark.org> > wrote:
> 
> 
> 
> 
>   > -Original Message-
>   > From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org
> <mailto:wireshark-dev-boun...@wireshark.org> ] On Behalf
>   > Of Jeff Morriss
>   > Sent: Friday, July 14, 2017 10:49 AM
>   > To: Developer support list for Wireshark  d...@wireshark.org <mailto:wireshark-dev@wireshark.org> >
>   > Subject: Re: [Wireshark-dev] Fields offsets & tree hierarchy questions
>   >
>   >
>   >
>   > On Fri, Jul 14, 2017 at 1:02 PM, Sultan, Hassan via Wireshark-dev
>> d...@wireshark.org <mailto:d...@wireshark.org>  <mailto:wireshark-
> d...@wireshark.org <mailto:wireshark-dev@wireshark.org> > > wrote:
>   >
>   >
>   [...]
>   >
>   >   2) When looking at http.file_data(65), the field's offset is 0, 
> relative
> to
>   > that field's tvb which contains the decompressed data, is there any
> way to get
>   > the position relative to the 'main' tvb representing the whole packet 
> ?
> I couldn't
>   > find one but maybe I'm missing something. http.file_data(65)
> represents
>   > decompressed data so technically not present in the main tvb, but I
> was
>   > wondering if there was a way to link it to the compressed data field 
> it
> represents
>   > (the "text(83)" field)
>   >
>   >
>   >
>   > I don't think so.  Only the HTTP dissector (and a human) know that the
> dissector
>   > created a new TVB out of data from the text(83) field.  There's no
> linkage (that I
>   > recall) between those 2 TVBs (unlike, say, subset TVBs).
> 
>   I see. I'm trying to figure out how automation could make the
> difference so it can ignore fields like this one that represent the same data 
> as
> another field.
> 
>   One way would be to look for fields that contain an ft_value , though
> this would mean some segments of the packet_data would be missing
> (http.response.code being an example, there's no field without an ft_value
> representing those bytes of the packet), or maybe ignore any field that has
> offset 0 without being the top-level field ? It seems a bit hackish though.
> 
> 
> 
> Maybe there's a way by comparing proto_node->field_info->ds_tvb?  For such
> packets there will be 2 (or more) TVBs.

Great idea ! I'll give it a try and see what I get.

Thanks !

Hassan

___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Fields offsets & tree hierarchy questions

2017-07-14 Thread Jeff Morriss 
On Fri, Jul 14, 2017 at 2:01 PM, Sultan, Hassan via Wireshark-dev <
wireshark-dev@wireshark.org> wrote:

>
>
> > -Original Message-
> > From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On
> Behalf
> > Of Jeff Morriss
> > Sent: Friday, July 14, 2017 10:49 AM
> > To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
> > Subject: Re: [Wireshark-dev] Fields offsets & tree hierarchy questions
> >
> >
> >
> > On Fri, Jul 14, 2017 at 1:02 PM, Sultan, Hassan via Wireshark-dev
>  > d...@wireshark.org <mailto:wireshark-dev@wireshark.org> > wrote:
> >
> >
> [...]
> >
> >   2) When looking at http.file_data(65), the field's offset is 0,
> relative to
> > that field's tvb which contains the decompressed data, is there any way
> to get
> > the position relative to the 'main' tvb representing the whole packet ?
> I couldn't
> > find one but maybe I'm missing something. http.file_data(65) represents
> > decompressed data so technically not present in the main tvb, but I was
> > wondering if there was a way to link it to the compressed data field it
> represents
> > (the "text(83)" field)
> >
> >
> >
> > I don't think so.  Only the HTTP dissector (and a human) know that the
> dissector
> > created a new TVB out of data from the text(83) field.  There's no
> linkage (that I
> > recall) between those 2 TVBs (unlike, say, subset TVBs).
>
> I see. I'm trying to figure out how automation could make the difference
> so it can ignore fields like this one that represent the same data as
> another field.
>
> One way would be to look for fields that contain an ft_value , though this
> would mean some segments of the packet_data would be missing
> (http.response.code being an example, there's no field without an ft_value
> representing those bytes of the packet), or maybe ignore any field that has
> offset 0 without being the top-level field ? It seems a bit hackish though.
>

Maybe there's a way by comparing proto_node->field_info->ds_tvb?  For such
packets there will be 2 (or more) TVBs.
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Fields offsets & tree hierarchy questions

2017-07-14 Thread Sultan, Hassan via Wireshark-dev 


> -Original Message-
> From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf
> Of Jeff Morriss
> Sent: Friday, July 14, 2017 10:49 AM
> To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
> Subject: Re: [Wireshark-dev] Fields offsets & tree hierarchy questions
> 
> 
> 
> On Fri, Jul 14, 2017 at 1:02 PM, Sultan, Hassan via Wireshark-dev  d...@wireshark.org <mailto:wireshark-dev@wireshark.org> > wrote:
> 
> 
[...]
> 
>   2) When looking at http.file_data(65), the field's offset is 0, 
> relative to
> that field's tvb which contains the decompressed data, is there any way to get
> the position relative to the 'main' tvb representing the whole packet ? I 
> couldn't
> find one but maybe I'm missing something. http.file_data(65) represents
> decompressed data so technically not present in the main tvb, but I was
> wondering if there was a way to link it to the compressed data field it 
> represents
> (the "text(83)" field)
> 
> 
> 
> I don't think so.  Only the HTTP dissector (and a human) know that the 
> dissector
> created a new TVB out of data from the text(83) field.  There's no linkage 
> (that I
> recall) between those 2 TVBs (unlike, say, subset TVBs).

I see. I'm trying to figure out how automation could make the difference so it 
can ignore fields like this one that represent the same data as another field.

One way would be to look for fields that contain an ft_value , though this 
would mean some segments of the packet_data would be missing 
(http.response.code being an example, there's no field without an ft_value 
representing those bytes of the packet), or maybe ignore any field that has 
offset 0 without being the top-level field ? It seems a bit hackish though.

Thanks,

Hassan
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Fields offsets & tree hierarchy questions

2017-07-14 Thread Jeff Morriss 
On Fri, Jul 14, 2017 at 1:02 PM, Sultan, Hassan via Wireshark-dev <
wireshark-dev@wireshark.org> wrote:

> Hi everyone,
>
> Sorry to bother you with might be beginner questions but... well... I'm a
> beginner :)
>
> In my quest to understand how Wireshark's parsing engine works I've
> written a small wrapper that iterates through all parsed fields and
> displays them in the following format :
>
> [offset] [abbrev]([length]) : [value, which might be either my
> interpretation of the bytes on the wire according to type/encoding info for
> the types I currently support, or the ft_value of the field if it is
> present]
>
> And for some packets I am getting :
>

[...]


> Which brings the following questions :
>
> 1) Am I right to assume there are absolutely no guarantee about the order
> of fields reported by proto_tree_children_foreach in regard to offset
> within the same tvb ? I'm looking at the json fields and the offsets of
> sub-fields are not ordered. I guess the order is the order in which the
> fields were added to the tree ?
>

Yes, that would be my expectation.  Fields are reported (in the UI and
other places) in the order that they were added to the tree.


> 2) When looking at http.file_data(65), the field's offset is 0, relative
> to that field's tvb which contains the decompressed data, is there any way
> to get the position relative to the 'main' tvb representing the whole
> packet ? I couldn't find one but maybe I'm missing something.
> http.file_data(65) represents decompressed data so technically not present
> in the main tvb, but I was wondering if there was a way to link it to the
> compressed data field it represents (the "text(83)" field)
>

I don't think so.  Only the HTTP dissector (and a human) know that the
dissector created a new TVB out of data from the text(83) field.  There's
no linkage (that I recall) between those 2 TVBs (unlike, say, subset TVBs).
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Fields offsets & tree hierarchy questions

2017-07-14 Thread Sultan, Hassan via Wireshark-dev 
Nevermind the last question, that was me being dumb and fooled by the offset. 
They actually are under the http tree

> -Original Message-
> From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf
> Of Sultan, Hassan via Wireshark-dev
> Sent: Friday, July 14, 2017 10:03 AM
> To: wireshark-dev@wireshark.org
> Cc: Sultan, Hassan <sul...@amazon.com>
> Subject: [Wireshark-dev] Fields offsets & tree hierarchy questions
> 
> Hi everyone,
> 
> Sorry to bother you with might be beginner questions but... well... I'm a 
> beginner
> :)
> 
> In my quest to understand how Wireshark's parsing engine works I've written a
> small wrapper that iterates through all parsed fields and displays them in the
> following format :
> 
> [offset] [abbrev]([length]) : [value, which might be either my interpretation 
> of
> the bytes on the wire according to type/encoding info for the types I 
> currently
> support, or the ft_value of the field if it is present]
> 
> And for some packets I am getting :
> 
> 66 http(319) :
> 66 text(17) : 485454502F312E3120323030204F4B0D0A
> 66 http.request.version(8) : HTTP/1.1
> 75 http.response.code(3) : 200
> 79 http.response.phrase(2) : OK
> 83 http.response.line(44) : Access-Control-Allow-Headers: content-type
> 
> 127 http.response.line(32) : Access-Control-Allow-Origin: *
> 
> 159 http.content_encoding(24) : gzip
> 183 http.content_type(32) : application/json
> 215 http.date(37) : Thu, 13 Jul 2017 23:07:22 GMT
> 252 http.server(19) : openresty
> 271 http.response.line(23) : Vary: Accept-Encoding
> 
> 294 http.response.line(16) : X-Cache-Hit: 0
> 
> 310 http.response.line(29) : X-Frame-Options: SAMEORIGIN
> 
> 339 http.content_length_header(20) : 83
> 359 http.connection(24) : keep-alive
> 383 text(2) : 0D0A
> 385 text(83) :
> 1F8B080305C1C10E80200800D07FE11C5B5349F2671A226E5D25
> 4EAD7FEFBD17C26D5DF7800647B72A3A0B4AE689599490B9EE483258E5A4222
> 9C1061EAAE60EED5961DF0FC6434ECF4100
> 0 http.file_data(65) : {"user_id":"6be7acf4-a38f-3ac5-8870-
> 5ad8ca954a22","success":true}
> 0 json(65) :
> 0 json.object(65) :
> 0E1827559C340E664E8DFFAE0800450001C6999B4000EA06847F364D8BBFAC1F
> 02EB0050D864F3E601114594EC358018007DEF250101080A46DA716F0405
> 41
> 1 json.member(48) :
> 1827559C340E664E8DFFAE0800450001C6999B4000EA06847F364D8BBFAC1F02
> EB0050D864F3E601114594EC35801800
> 11 json.value.string(38) : 
> 6be7acf4-a38f-3ac5-8870-5ad8ca954a22
> 1 json.key(9) : user_id
> 50 json.member(14) : EF250101080A46DA716F0405
> 60 json.value.true(4) : 716F0405
> 50 json.key(9) : success
> 
> Which brings the following questions :
> 
> 1) Am I right to assume there are absolutely no guarantee about the order of
> fields reported by proto_tree_children_foreach in regard to offset within the
> same tvb ? I'm looking at the json fields and the offsets of sub-fields are 
> not
> ordered. I guess the order is the order in which the fields were added to the 
> tree
> ?
> 2) When looking at http.file_data(65), the field's offset is 0, relative to 
> that
> field's tvb which contains the decompressed data, is there any way to get the
> position relative to the 'main' tvb representing the whole packet ? I 
> couldn't find
> one but maybe I'm missing something. http.file_data(65) represents
> decompressed data so technically not present in the main tvb, but I was
> wondering if there was a way to link it to the compressed data field it 
> represents
> (the "text(83)" field)
> 3) I'm curious to know why the "text(83)" field and "json.object(65)" fields 
> are
> not under the http tree, that's where I would have expected them to be
> 
> Thanks for your help,
> 
> Hassan
> _
> __
> Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
>  mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

[Wireshark-dev] Fields offsets & tree hierarchy questions

2017-07-14 Thread Sultan, Hassan via Wireshark-dev 
Hi everyone,

Sorry to bother you with might be beginner questions but... well... I'm a 
beginner :)

In my quest to understand how Wireshark's parsing engine works I've written a 
small wrapper that iterates through all parsed fields and displays them in the 
following format :

[offset] [abbrev]([length]) : [value, which might be either my interpretation 
of the bytes on the wire according to type/encoding info for the types I 
currently support, or the ft_value of the field if it is present]

And for some packets I am getting :

66 http(319) : 
66 text(17) : 485454502F312E3120323030204F4B0D0A
66 http.request.version(8) : HTTP/1.1
75 http.response.code(3) : 200
79 http.response.phrase(2) : OK
83 http.response.line(44) : Access-Control-Allow-Headers: content-type

127 http.response.line(32) : Access-Control-Allow-Origin: *

159 http.content_encoding(24) : gzip
183 http.content_type(32) : application/json
215 http.date(37) : Thu, 13 Jul 2017 23:07:22 GMT
252 http.server(19) : openresty
271 http.response.line(23) : Vary: Accept-Encoding

294 http.response.line(16) : X-Cache-Hit: 0

310 http.response.line(29) : X-Frame-Options: SAMEORIGIN

339 http.content_length_header(20) : 83
359 http.connection(24) : keep-alive
383 text(2) : 0D0A
385 text(83) : 
1F8B080305C1C10E80200800D07FE11C5B5349F2671A226E5D254EAD7FEFBD17C26D5DF7800647B72A3A0B4AE689599490B9EE483258E5A42229C1061EAAE60EED5961DF0FC6434ECF4100
0 http.file_data(65) : 
{"user_id":"6be7acf4-a38f-3ac5-8870-5ad8ca954a22","success":true}
0 json(65) : 
0 json.object(65) : 
0E1827559C340E664E8DFFAE0800450001C6999B4000EA06847F364D8BBFAC1F02EB0050D864F3E601114594EC358018007DEF250101080A46DA716F040541
1 json.member(48) : 
1827559C340E664E8DFFAE0800450001C6999B4000EA06847F364D8BBFAC1F02EB0050D864F3E601114594EC35801800
11 json.value.string(38) : 
6be7acf4-a38f-3ac5-8870-5ad8ca954a22
1 json.key(9) : user_id
50 json.member(14) : EF250101080A46DA716F0405
60 json.value.true(4) : 716F0405
50 json.key(9) : success

Which brings the following questions :

1) Am I right to assume there are absolutely no guarantee about the order of 
fields reported by proto_tree_children_foreach in regard to offset within the 
same tvb ? I'm looking at the json fields and the offsets of sub-fields are not 
ordered. I guess the order is the order in which the fields were added to the 
tree ?
2) When looking at http.file_data(65), the field's offset is 0, relative to 
that field's tvb which contains the decompressed data, is there any way to get 
the position relative to the 'main' tvb representing the whole packet ? I 
couldn't find one but maybe I'm missing something. http.file_data(65) 
represents decompressed data so technically not present in the main tvb, but I 
was wondering if there was a way to link it to the compressed data field it 
represents (the "text(83)" field)
3) I'm curious to know why the "text(83)" field and "json.object(65)" fields 
are not under the http tree, that's where I would have expected them to be

Thanks for your help,

Hassan
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe