Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Paul Offord]

I've realised below is a stupid idea.  I was trying to be clever by picking 
numbers that spelt out the block type in a hex dump of a file, but I've 
realised this is of little use.  I'll use:

•   0x0010 – Text Source Descriptor Block (TSDB)
•   0x0011 – Text Source Record Block (TRB)

This leaves some space for base types.

Best regards...Paul

-Original Message-
From: Paul Offord 
Sent: 25 July 2017 10:36
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: RE: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

Hi,

Sorry - Slight amendment.

Can I register two new PCAP-NG block codes?  I’d like:

•   0x42445354 – Text Source Descriptor Block (TSDB)
•   0x00425254 – Text Source Record Block (TRB)

What do I need to do to register these?

See below for the reason for request.

Thanks and regards…Paul

-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Paul Offord
Sent: 25 July 2017 10:26
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

Hi,

Can I register two new PCAP-NG block codes?  I’d like:

•   0x54534442 – Text Source Descriptor Block (TSDB)
•   0x54524200 – Text Source Record Block (TRB)

What do I need to do to register these?

See below for the reason for request.

Thanks and regards…Paul


-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Guy Harris
Sent: 19 June 2017 22:54
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

On Jun 19, 2017, at 3:06 AM, Paul Offord <paul.off...@advance7.com> wrote:

>> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
>> logs that are in CSV and TSV format"?
> 
> Yes - I mean arbitrary comma, tab and space separated.

I think Microsoft have an Excellent program that reads CSV files - and possibly 
TSV and SSV as well - and lets you do various forms of processing on them; it's 
available for Windows and macOS.  I think there's another program called 
LibreOffice Calc that's available for those platforms and for Linux, *BSD, and 
other OSes as well.

I.e., what does *Wireshark* provide here, for the *general* case of CSV/TSV/SSV 
files, that a spreadsheet program doesn't?

> Going back to the main objective, what I'm proposing is that we enable a 
> PCAP-NG file to contain any text-based data together with a schema - JSON or 
> XML or PCAP-NG proprietary.  There would be one WTAP_ENCAP_xxx type for any 
> text-based data type. Decoding would be based on the schema data.

pcapng, unlike pcap, can have blocks in it that *don't* correspond to packets.  
What's needed here are new pcapng block types:

a "text source description block" (TSDB), which contains a schema and 
options (such as comments);

a "text record block" (TRB), which contains:

a "text source ID", which indicates which TSDB corresponds to 
the source of the record (the first TSDB has an ID of 0, the next one an ID of 
1, etc., along the lines of Interface Description Blocks);

the text;

options (such as comments).

This would *not* require a DLT_/LINKTYPE_ or a WTAP_ENCAP_ value.  It would 
also allow packets and text records to be combined into one file.
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

__

This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and delete 
this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not 
necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour 
House, Coopers End Lane, Stansted, Essex CM24 1SJ

Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Paul Offord]

Hi,

Sorry - Slight amendment.

Can I register two new PCAP-NG block codes?  I’d like:

•   0x42445354 – Text Source Descriptor Block (TSDB)
•   0x00425254 – Text Source Record Block (TRB)

What do I need to do to register these?

See below for the reason for request.

Thanks and regards…Paul

-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Paul Offord
Sent: 25 July 2017 10:26
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

Hi,

Can I register two new PCAP-NG block codes?  I’d like:

•   0x54534442 – Text Source Descriptor Block (TSDB)
•   0x54524200 – Text Source Record Block (TRB)

What do I need to do to register these?

See below for the reason for request.

Thanks and regards…Paul


-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Guy Harris
Sent: 19 June 2017 22:54
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

On Jun 19, 2017, at 3:06 AM, Paul Offord <paul.off...@advance7.com> wrote:

>> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
>> logs that are in CSV and TSV format"?
> 
> Yes - I mean arbitrary comma, tab and space separated.

I think Microsoft have an Excellent program that reads CSV files - and possibly 
TSV and SSV as well - and lets you do various forms of processing on them; it's 
available for Windows and macOS.  I think there's another program called 
LibreOffice Calc that's available for those platforms and for Linux, *BSD, and 
other OSes as well.

I.e., what does *Wireshark* provide here, for the *general* case of CSV/TSV/SSV 
files, that a spreadsheet program doesn't?

> Going back to the main objective, what I'm proposing is that we enable a 
> PCAP-NG file to contain any text-based data together with a schema - JSON or 
> XML or PCAP-NG proprietary.  There would be one WTAP_ENCAP_xxx type for any 
> text-based data type. Decoding would be based on the schema data.

pcapng, unlike pcap, can have blocks in it that *don't* correspond to packets.  
What's needed here are new pcapng block types:

a "text source description block" (TSDB), which contains a schema and 
options (such as comments);

a "text record block" (TRB), which contains:

a "text source ID", which indicates which TSDB corresponds to 
the source of the record (the first TSDB has an ID of 0, the next one an ID of 
1, etc., along the lines of Interface Description Blocks);

the text;

options (such as comments).

This would *not* require a DLT_/LINKTYPE_ or a WTAP_ENCAP_ value.  It would 
also allow packets and text records to be combined into one file.
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

__

This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and delete 
this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not 
necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour 
House, Coopers End Lane, Stansted, Essex CM24 1SJ

__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com 
__
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wires

Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Paul Offord]

Hi,

Can I register two new PCAP-NG block codes?  I’d like:

•   0x54534442 – Text Source Descriptor Block (TSDB)
•   0x54524200 – Text Source Record Block (TRB)

What do I need to do to register these?

See below for the reason for request.

Thanks and regards…Paul


-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Guy Harris
Sent: 19 June 2017 22:54
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

On Jun 19, 2017, at 3:06 AM, Paul Offord <paul.off...@advance7.com> wrote:

>> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
>> logs that are in CSV and TSV format"?
> 
> Yes - I mean arbitrary comma, tab and space separated.

I think Microsoft have an Excellent program that reads CSV files - and possibly 
TSV and SSV as well - and lets you do various forms of processing on them; it's 
available for Windows and macOS.  I think there's another program called 
LibreOffice Calc that's available for those platforms and for Linux, *BSD, and 
other OSes as well.

I.e., what does *Wireshark* provide here, for the *general* case of CSV/TSV/SSV 
files, that a spreadsheet program doesn't?

> Going back to the main objective, what I'm proposing is that we enable a 
> PCAP-NG file to contain any text-based data together with a schema - JSON or 
> XML or PCAP-NG proprietary.  There would be one WTAP_ENCAP_xxx type for any 
> text-based data type. Decoding would be based on the schema data.

pcapng, unlike pcap, can have blocks in it that *don't* correspond to packets.  
What's needed here are new pcapng block types:

a "text source description block" (TSDB), which contains a schema and 
options (such as comments);

a "text record block" (TRB), which contains:

a "text source ID", which indicates which TSDB corresponds to 
the source of the record (the first TSDB has an ID of 0, the next one an ID of 
1, etc., along the lines of Interface Description Blocks);

the text;

options (such as comments).

This would *not* require a DLT_/LINKTYPE_ or a WTAP_ENCAP_ value.  It would 
also allow packets and text records to be combined into one file.
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

__

This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and delete 
this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not 
necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour 
House, Coopers End Lane, Stansted, Essex CM24 1SJ

__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Paul Offord]

Or to think of it another way, a feature to allow:

* The merging (enrichment) of network data with other data types - particularly 
attractive in these days of increasing encryption
* Extending the availability of the great features of Wireshark into other 
areas of IT support

Best regards...Paul

-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Jaap Keuter
Sent: 19 June 2017 07:43
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

Hi,

So is this an attempt to make Wireshark into a glorified log file viewer? Kinda 
like Maslow's hammer?

Thanks,
Jaap


> On 19 Jun 2017, at 12:06, Paul Offord <paul.off...@advance7.com> wrote:
> 
>> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
>> logs that are in CSV and TSV format"?
> 
> Yes - I mean arbitrary comma, tab and space separated.  IIS web logs are in 
> Space Separated Variable format (standard W3 format I believe).
> 
>> And is there any reason not to just take the log file reading part of your 
>> code, make it into a libwiretap module that *directly* reads those files 
>> using a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
>> WTAP_ENCAP_IIS_LOG?
> 
> I'm sure it's possible, and that would be a future project.  The code is 
> currently a Win32 Console app written in C++.
> 
>> Note that this would require support for random access to the file when 
>> reading it.
> 
> Yes, and a whole lot more.  I'm going to extend my console app to 
> automatically determine data types, and we would need to replicate this in 
> Wireshark if we are to read the files directly.  I think there will be  whole 
> bunch more challenges, and so you can see that integrating this code into 
> Wireshark is going to be quite a task.
> 
> Going back to the main objective, what I'm proposing is that we enable a 
> PCAP-NG file to contain any text-based data together with a schema - JSON or 
> XML or PCAP-NG proprietary.  There would be one WTAP_ENCAP_xxx type for any 
> text-based data type.  Decoding would be based on the schema data.
> 
> Best regards...Paul
> 
> -Original Message-
> From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On 
> Behalf Of Guy Harris
> Sent: 18 June 2017 23:01
> To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
> Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV 
> and SSV data types
> 
> On Jun 18, 2017, at 5:54 PM, Paul Offord <paul.off...@advance7.com> wrote:
> 
>> Some time ago I wrote a program that reads Microsoft IIS web log files and 
>> converts them into PCAP-NG format.  I then wrote a matching dissector to 
>> produce nice neat output.  I want to rewrite it because:
>> 
>>  • The field names and data types are defined in the first record of the 
>> PCAP-NG file – this is how the dissector knows the layout of the file
>>  • Each data record is encapsulated in a dummy Ethernet frame and I want 
>> to eliminate this
>>  • I’d like to make it more general so that it can also deal with 
>> CSV, TSV and, later, variable format log records
> 
> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
> logs that are in CSV and TSV format"?
> 
> And is there any reason not to just take the log file reading part of your 
> code, make it into a libwiretap module that *directly* reads those files 
> using a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
> WTAP_ENCAP_IIS_LOG?
> 
> Note that this would require support for random access to the file when 
> reading it.
> ___
> Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
> 
> mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
> 
> __
> 
> This message contains confidential information and is intended only for the 
> individual named. If you are not the named addressee you should not 
> disseminate, distribute or copy this e-mail. Please notify the sender 
> immediately by e-mail if you have received this e-mail by mistake and delete 
> this e-mail from your system.
> 
> Any views or opinions expressed are solely those of the author and do not 
> necessarily represent those of Advance Seven Ltd. E-mail transmission cannot 
&

Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Jaap Keuter]

Hi,

So is this an attempt to make Wireshark into a glorified log file viewer? Kinda 
like Maslow's hammer?

Thanks,
Jaap


> On 19 Jun 2017, at 12:06, Paul Offord <paul.off...@advance7.com> wrote:
> 
>> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
>> logs that are in CSV and TSV format"?
> 
> Yes - I mean arbitrary comma, tab and space separated.  IIS web logs are in 
> Space Separated Variable format (standard W3 format I believe).
> 
>> And is there any reason not to just take the log file reading part of your 
>> code, make it into a libwiretap module that *directly* reads those files 
>> using a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
>> WTAP_ENCAP_IIS_LOG?
> 
> I'm sure it's possible, and that would be a future project.  The code is 
> currently a Win32 Console app written in C++.
> 
>> Note that this would require support for random access to the file when 
>> reading it.
> 
> Yes, and a whole lot more.  I'm going to extend my console app to 
> automatically determine data types, and we would need to replicate this in 
> Wireshark if we are to read the files directly.  I think there will be  whole 
> bunch more challenges, and so you can see that integrating this code into 
> Wireshark is going to be quite a task.
> 
> Going back to the main objective, what I'm proposing is that we enable a 
> PCAP-NG file to contain any text-based data together with a schema - JSON or 
> XML or PCAP-NG proprietary.  There would be one WTAP_ENCAP_xxx type for any 
> text-based data type.  Decoding would be based on the schema data.
> 
> Best regards...Paul
> 
> -Original Message-
> From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
> Guy Harris
> Sent: 18 June 2017 23:01
> To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
> Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
> data types
> 
> On Jun 18, 2017, at 5:54 PM, Paul Offord <paul.off...@advance7.com> wrote:
> 
>> Some time ago I wrote a program that reads Microsoft IIS web log files and 
>> converts them into PCAP-NG format.  I then wrote a matching dissector to 
>> produce nice neat output.  I want to rewrite it because:
>> 
>>  • The field names and data types are defined in the first record of the 
>> PCAP-NG file – this is how the dissector knows the layout of the file
>>  • Each data record is encapsulated in a dummy Ethernet frame and I want 
>> to eliminate this
>>  • I’d like to make it more general so that it can also deal with CSV, 
>> TSV and, later, variable format log records
> 
> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
> logs that are in CSV and TSV format"?
> 
> And is there any reason not to just take the log file reading part of your 
> code, make it into a libwiretap module that *directly* reads those files 
> using a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
> WTAP_ENCAP_IIS_LOG?
> 
> Note that this would require support for random access to the file when 
> reading it.
> ___
> Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
> Archives:https://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
> mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe
> 
> __
> 
> This message contains confidential information and is intended only for the 
> individual named. If you are not the named addressee you should not 
> disseminate, distribute or copy this e-mail. Please notify the sender 
> immediately by e-mail if you have received this e-mail by mistake and delete 
> this e-mail from your system.
> 
> Any views or opinions expressed are solely those of the author and do not 
> necessarily represent those of Advance Seven Ltd. E-mail transmission cannot 
> be guaranteed to be secure or error-free as information could be intercepted, 
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
> The sender therefore does not accept liability for any errors or omissions in 
> the contents of this message, which arise as a result of e-mail transmission.
> 
> Advance Seven Ltd. Registered in England & Wales numbered 2373877 at 
> Endeavour House, Coopers End Lane, Stansted, Essex CM24 1SJ
> 
> __
> This email has been scanned by the Symantec Ema

Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Paul Offord]

> Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web 
> logs that are in CSV and TSV format"?

Yes - I mean arbitrary comma, tab and space separated.  IIS web logs are in 
Space Separated Variable format (standard W3 format I believe).

> And is there any reason not to just take the log file reading part of your 
> code, make it into a libwiretap module that *directly* reads those files 
> using a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
> WTAP_ENCAP_IIS_LOG?

I'm sure it's possible, and that would be a future project.  The code is 
currently a Win32 Console app written in C++.

> Note that this would require support for random access to the file when 
> reading it.

Yes, and a whole lot more.  I'm going to extend my console app to automatically 
determine data types, and we would need to replicate this in Wireshark if we 
are to read the files directly.  I think there will be  whole bunch more 
challenges, and so you can see that integrating this code into Wireshark is 
going to be quite a task.

Going back to the main objective, what I'm proposing is that we enable a 
PCAP-NG file to contain any text-based data together with a schema - JSON or 
XML or PCAP-NG proprietary.  There would be one WTAP_ENCAP_xxx type for any 
text-based data type.  Decoding would be based on the schema data.

Best regards...Paul

-Original Message-
From: Wireshark-dev [mailto:wireshark-dev-boun...@wireshark.org] On Behalf Of 
Guy Harris
Sent: 18 June 2017 23:01
To: Developer support list for Wireshark <wireshark-dev@wireshark.org>
Subject: Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV 
data types

On Jun 18, 2017, at 5:54 PM, Paul Offord <paul.off...@advance7.com> wrote:

> Some time ago I wrote a program that reads Microsoft IIS web log files and 
> converts them into PCAP-NG format.  I then wrote a matching dissector to 
> produce nice neat output.  I want to rewrite it because:
>  
>   • The field names and data types are defined in the first record of the 
> PCAP-NG file – this is how the dissector knows the layout of the file
>   • Each data record is encapsulated in a dummy Ethernet frame and I want 
> to eliminate this
>   • I’d like to make it more general so that it can also deal with CSV, 
> TSV and, later, variable format log records

Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web logs 
that are in CSV and TSV format"?

And is there any reason not to just take the log file reading part of your 
code, make it into a libwiretap module that *directly* reads those files using 
a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
WTAP_ENCAP_IIS_LOG?

Note that this would require support for random access to the file when reading 
it.
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

__

This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and delete 
this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not 
necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour 
House, Coopers End Lane, Stansted, Essex CM24 1SJ

__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
__
___
Sent via:Wireshark-dev mailing list <wireshark-dev@wireshark.org>
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

Re: [Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Guy Harris]

On Jun 18, 2017, at 5:54 PM, Paul Offord  wrote:

> Some time ago I wrote a program that reads Microsoft IIS web log files and 
> converts them into PCAP-NG format.  I then wrote a matching dissector to 
> produce nice neat output.  I want to rewrite it because:
>  
>   • The field names and data types are defined in the first record of the 
> PCAP-NG file – this is how the dissector knows the layout of the file
>   • Each data record is encapsulated in a dummy Ethernet frame and I want 
> to eliminate this
>   • I’d like to make it more general so that it can also deal with CSV, 
> TSV and, later, variable format log records

Do you mean "deal with arbitrary CSV and TSV files" or "deal with IIS Web logs 
that are in CSV and TSV format"?

And is there any reason not to just take the log file reading part of your 
code, make it into a libwiretap module that *directly* reads those files using 
a WTAP_ENCAP_IIS_LOG encapsulation, and write a dissector for 
WTAP_ENCAP_IIS_LOG?

Note that this would require support for random access to the file when reading 
it.
___
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe

[Wireshark-dev] Adding support to Wireshark for CSV, TSV and SSV data types

[Paul Offord]

Some time ago I wrote a program that reads Microsoft IIS web log files and 
converts them into PCAP-NG format.  I then wrote a matching dissector to 
produce nice neat output.  I want to rewrite it because:


  1.  The field names and data types are defined in the first record of the 
PCAP-NG file - this is how the dissector knows the layout of the file
  2.  Each data record is encapsulated in a dummy Ethernet frame and I want to 
eliminate this
  3.  I'd like to make it more general so that it can also deal with CSV, TSV 
and, later, variable format log records

My plan is to:


  1.  Insert the field name and data type information into a new IDB option 
block
  2.  Write a plugin_if function to enable a dissector to get a pointer to the 
IDB information and hence get my hands on the field names and data types
  3.  Define a new link type for the IDB; working name is Text Data Format (TDF)
 *   I understand that I have to ask for this (a value for DLT_TDF) from 
the tcpdump-work...@lists.tcpdump.org
 *   This means that the TDF records would be below (within) the Frame
  4.  Add a new WTAP_ENCAP_TDF definition in wtap.h
  5.  Add an appropriate entry to the pcap_to_wtap_map[] array in pcap-common.c
  6.  Write a dissector for the TDF records (dissect_tdf)
  7.  Use dissector_add_uint("wtap_encap", WTAP_ENCAP_TDF, tdf_handle) to hook 
the dissector into the correct point in the table

Does this look like a reasonable plan?

Have a missed an existing feature that I should be using?  I remember seeing 
Graham Bloises presentation on three ways to write dissectors; C, LUA and 
something else.  Wasn't the something else about using a high-level description 
of the packets.  I'd look it up but I'm on a plane at the moment.

Thanks and regards...Paul

PS:  Bit of an update.  Just made the changes to eliminate the Ethernet headers 
(with dummy DLT_TDF value), modified my dissector and it works!  Wow - that was 
a lot simpler than I thought.


__

This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and delete 
this e-mail from your system.

Any views or opinions expressed are solely those of the author and do not 
necessarily represent those of Advance Seven Ltd. E-mail transmission cannot be 
guaranteed to be secure or error-free as information could be intercepted, 
corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message, which arise as a result of e-mail transmission.

Advance Seven Ltd. Registered in England & Wales numbered 2373877 at Endeavour 
House, Coopers End Lane, Stansted, Essex CM24 1SJ

__
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
_
Sent via:Wireshark-dev mailing list 
Archives:https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
 mailto:wireshark-dev-requ...@wireshark.org?subject=unsubscribe