Re: [netsniff-ng] Few Questions I could not find answer for :)

2016-12-05 Thread Vadim Kochan
On Tue, Dec 6, 2016 at 2:19 AM, Shaked, Asaf  wrote:
> Hi,
>
>
>
> Sorry for not responding earlier, I was buried with work J
>
> I’ll try to explain in more details what I was trying to accomplish…
>
>
>
>
>
> After using Netsniff-NG to record high throughput traffic impressively, I
> was trying it as capture Replayer.
>
>
>
> I was trying to replay, in my performance tuned local lab, high throughput
> traffic, recorded on remote site, using some expensive equipment.
>
>
>
> I received the recording split to multiple pieces, with following details:
> (capinfos)
>
> File type:   Wireshark/tcpdump/... - pcap
>
> File encapsulation:  Ethernet
>
> Packet size limit:   file hdr: 9000 bytes
>
> Number of packets:   6382 k
>
> File size:   4294 MB
>
> Data size:   4192 MB
>
> Capture duration:5 seconds
>
> Data byte rate:  824 MBps
>
> Data bit rate:   6592 Mbps
>
> Average packet size: 656.97 bytes
>
> Average packet rate: 1254 kpackets/sec
>
> Strict time order:   True
>
>
>
> And I also left MergeCap to run through the night, to have also as single
> capture, with following details:
>
> File type:   Wireshark/tcpdump/... - pcap
>
> File encapsulation:  Ethernet
>
> Packet size limit:   file hdr: 9000 bytes
>
> Number of packets:   1348 M
>
> File size:   905 GB
>
> Data size:   883 GB
>
> Capture duration:1096 seconds
>
> Data byte rate:  806 MBps
>
> Data bit rate:   6451 Mbps
>
> Average packet size: 655.45 bytes
>
> Average packet rate: 1230 kpackets/sec
>
> Strict time order:   False
>
>
>
> 1)  Notice I lost the “Strict time order” in merged file – don’t
> understand why :\
>
>
>
> VM I set up for replay is 64bit RHEL 7.2 with 16 x vCores, 64GB RAM, VMXNET3
> adapter & 1TB on 10K SAS FC NetApp Storage,
>
> Hosted on Dell FC630, with Intel Xeon CPU E5-2670 v3 @ 2.3GHz & Intel X710
> 10g NIC
>
>
>
> So, the first (optimistic J) thing I tried to do is replay the merged file
> using Netsniff-NG…
>
> netsniff-ng -i .pcap -m -o  -s -J
>
>
>
> But I quickly learned that Netsniff-NG loads the entire capture to RAM
> before starting to replay J
>
>
>
> 2)  Is it possible to tell Netsniff-NG not to do that ? so I can replay
> a capture much larger than available RAM (as eventually done with tcpreplay
> on ~850GB merged)
>
>

As I suppose to have this as a new feature in trafgen then I think we
may use the ring size value
from the -S,--ring-size option to check if it is filled by for example
80% then it is needed to load the next chunk
on the background.

>
> 3)  In addition, if Netsniff-NG would have the ability to receive a list
> of captures to replay consecutively without drops in between,
>
> I would be able to replay my ~4GB multiple pieces, without having to merge
> them, nor be unable to play merged because of insufficient RAM.
>
> I thought of rather simple design in which two files are loaded to memory
> concurrently throughout replay…
>
> Starting with 1&2 à as soon as 1 ends replay, 3 is loaded to memory à as
> soon as 2 ends replay, 4 is loaded to memory… and so on.

Hm, this sounds interesting, so the pcap files 1st will be sorted by
packet timestamps, and
only after loaded 1 by one on demand probably ...

>
>
>
> So, replaying list of multiple files is one thing, but I also had issues
> with replay speed.
>
>
>
> As you said Vadim, it seems Netsniff-NG replays any given capture, as fast
> as possible.
>
>
>
> 4)  So I noticed two related things when replayed single ~4GB piece…
>
> a.   Netsniff-NG turned single-threaded all of a sudden & was utilizing
> 100% of only a single core
>
> b.  Probably therefore, as fast as possible was not fast enough, as
> recorded 5 seconds in ~6.5Gbps, took 30 seconds to replay in ~1.2Gbps
>
>
>
> BTW, tcpreplay also started its’ replay at ~1.2Gbps (using --topspeed flag –
> see command used below), which gradually dropped to only ~250Mbps after 4+
> hours of replay.
>
> tcpreplay -i  --pktlen -t .pcap
>
Well, I am not sure how it would be possible to distribute packets
with a strict order between cpus.

>
>
> 5)  In other cases, where recording bitrates are more sane J I believe
> it would be rather powerful for Netsniff-NG to have tcpreplay speed related
> flags/features…
>
>
>
>-x, --multiplier=str   Modify replay speed to a given multiple
>
>-p, --pps=num  Replay packets at a given packets/sec
>
>-M, --mbps=str Replay packets at a given Mbps
>
>-t, --topspeed Replay packets as fast as possible
>
>
>
> While the default no parameters is rather original timestamps & bitrate.

As I said, I already have changes (which I did not send yet to review)
where I added possibility
to send packets from single pcap file in trafgen tool, and by default
it sends them at the rate
considering original packet's timestamps, but rate might be changed
via -b option.
I am not sure if it is needed to add new 

RE: [netsniff-ng] Few Questions I could not find answer for :)

2016-12-05 Thread Shaked, Asaf
Hi,

Sorry for not responding earlier, I was buried with work ☺
I’ll try to explain in more details what I was trying to accomplish…


After using Netsniff-NG to record high throughput traffic impressively, I was 
trying it as capture Replayer.

I was trying to replay, in my performance tuned local lab, high throughput 
traffic, recorded on remote site, using some expensive equipment.

I received the recording split to multiple pieces, with following details: 
(capinfos)
File type:   Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
Packet size limit:   file hdr: 9000 bytes
Number of packets:   6382 k
File size:   4294 MB
Data size:   4192 MB
Capture duration:5 seconds
Data byte rate:  824 MBps
Data bit rate:   6592 Mbps
Average packet size: 656.97 bytes
Average packet rate: 1254 kpackets/sec
Strict time order:   True

And I also left MergeCap to run through the night, to have also as single 
capture, with following details:
File type:   Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
Packet size limit:   file hdr: 9000 bytes
Number of packets:   1348 M
File size:   905 GB
Data size:   883 GB
Capture duration:1096 seconds
Data byte rate:  806 MBps
Data bit rate:   6451 Mbps
Average packet size: 655.45 bytes
Average packet rate: 1230 kpackets/sec
Strict time order:   False


1)  Notice I lost the “Strict time order” in merged file – don’t understand 
why :\

VM I set up for replay is 64bit RHEL 7.2 with 16 x vCores, 64GB RAM, VMXNET3 
adapter & 1TB on 10K SAS FC NetApp Storage,
Hosted on Dell FC630, with Intel Xeon CPU E5-2670 v3 @ 2.3GHz & Intel X710 10g 
NIC

So, the first (optimistic ☺) thing I tried to do is replay the merged file 
using Netsniff-NG…
netsniff-ng -i .pcap -m -o  -s -J

But I quickly learned that Netsniff-NG loads the entire capture to RAM before 
starting to replay ☺


2)  Is it possible to tell Netsniff-NG not to do that ? so I can replay a 
capture much larger than available RAM (as eventually done with tcpreplay on 
~850GB merged)


3)  In addition, if Netsniff-NG would have the ability to receive a list of 
captures to replay consecutively without drops in between,
I would be able to replay my ~4GB multiple pieces, without having to merge 
them, nor be unable to play merged because of insufficient RAM.
I thought of rather simple design in which two files are loaded to memory 
concurrently throughout replay…
Starting with 1&2 --> as soon as 1 ends replay, 3 is loaded to memory --> as 
soon as 2 ends replay, 4 is loaded to memory… and so on.

So, replaying list of multiple files is one thing, but I also had issues with 
replay speed.

As you said Vadim, it seems Netsniff-NG replays any given capture, as fast as 
possible.


4)  So I noticed two related things when replayed single ~4GB piece…

a.   Netsniff-NG turned single-threaded all of a sudden & was utilizing 
100% of only a single core

b.  Probably therefore, as fast as possible was not fast enough, as 
recorded 5 seconds in ~6.5Gbps, took 30 seconds to replay in ~1.2Gbps


BTW, tcpreplay also started its’ replay at ~1.2Gbps (using --topspeed flag – 
see command used below), which gradually dropped to only ~250Mbps after 4+ 
hours of replay.
tcpreplay -i  --pktlen -t .pcap


5)  In other cases, where recording bitrates are more sane ☺ I believe it 
would be rather powerful for Netsniff-NG to have tcpreplay speed related 
flags/features…



   -x, --multiplier=str   Modify replay speed to a given multiple

   -p, --pps=num  Replay packets at a given packets/sec

   -M, --mbps=str Replay packets at a given Mbps

   -t, --topspeed Replay packets as fast as possible



While the default no parameters is rather original timestamps & bitrate.



Would greatly appreciate your comments regarding any of the points ☺


Thanks,

Asaf.







-Original Message-
From: Vadim Kochan [mailto:vadi...@gmail.com]
Sent: יום ב 05 דצמבר 2016 11:57
To: Shaked, Asaf
Cc: netsniff-ng@googlegroups.com; Tobias Klauser; Daniel Borkmann
Subject: Re: [netsniff-ng] Few Questions I could not find answer for :)



Hi Tobias & Daniel,



What do you think is it worth to have it in trafgen with considering original 
timestamps by default ?



Thanks,



On Sun, Dec 4, 2016 at 6:28 PM, Vadim Kochan 
<vadi...@gmail.com<mailto:vadi...@gmail.com>> wrote:

> On Sun, Dec 4, 2016 at 2:09 PM, Vadim Kochan 
> <vadi...@gmail.com<mailto:vadi...@gmail.com>> wrote:

>> On Tue, Nov 22, 2016 at 10:32 AM, Shaked, Asaf 
>> <asaf.sha...@verint.com<mailto:asaf.sha...@verint.com>> wrote:

>>> Cool... good to see we're on the same page... thanks for the prompt

>>> response :)

>>>

>>> Asaf.

>>>

>>> -Original Message-

>>> From: Vadim Kochan [mailto:vadi.

Re: [netsniff-ng] Few Questions I could not find answer for :)

2016-12-05 Thread Vadim Kochan
Hi Tobias & Daniel,

What do you think is it worth to have it in trafgen with considering
original timestamps by default ?

Thanks,

On Sun, Dec 4, 2016 at 6:28 PM, Vadim Kochan <vadi...@gmail.com> wrote:
> On Sun, Dec 4, 2016 at 2:09 PM, Vadim Kochan <vadi...@gmail.com> wrote:
>> On Tue, Nov 22, 2016 at 10:32 AM, Shaked, Asaf <asaf.sha...@verint.com> 
>> wrote:
>>> Cool... good to see we're on the same page... thanks for the prompt 
>>> response :)
>>>
>>> Asaf.
>>>
>>> -Original Message-
>>> From: Vadim Kochan [mailto:vadi...@gmail.com]
>>> Sent: יום ג 22 נובמבר 2016 02:00
>>> To: Shaked, Asaf
>>> Cc: netsniff-ng@googlegroups.com
>>> Subject: Re: [netsniff-ng] Few Questions I could not find answer for :)
>>>
>>> On Mon, Nov 21, 2016 at 03:37:48PM +, Shaked, Asaf wrote:
>>> Hi Asaf,
>>>
>>>> Hi,
>>>>
>>>> Just love this toolkit :)
>>>>
>>>> Few questions, if you may...
>>>>
>>>> 1)  Is it possible to merge pcaps using the toolkit ?
>>> As fas as I know - no, but I think it would be good to have such feature 
>>> via such example:
>>>
>>> netsniff-ng -i {in_dir} -o {merged_file}.pcap
>>>
>>>>
>>>> 2)  Is it possible to replay a list of pcaps consecutively ?
>>>>
>>> As I know, it is posible only to redirect packets from one device to 
>>> another, but I really thing that it might be added to trafgen.
>>>
>>>> 3)  Is it possible to control pcaps replay bitrate ? (aiming for 
>>>> original capture average Bitrate, rather than "as fast as possible")
>>>>
>>> So if option #2 will be implemented then I think it wold be easy have it in 
>>> trafgen as there ia already rate limit option.
>>>
>>>> Thanks,
>>>>
>>>> Asaf.
>>>>
>>>
>>> Regards,
>>> Vadim Kochan
>>>
>>
>> Hi Asaf,
>>
>> I tried to extend trafgen with ability to send packets from pcap file,
>> and you can try it if you able to fetch & build trafgen from:
>>
>> https://github.com/vkochan/netsniff-ng/tree/trafgen_pcap_send
>>
>> By default original timestamps are considered when sending traffic
>> from pcap, but you
>> can reset sending rate via -b/-t options:
>>
>> sudo trafgen/trafgen -i /tmp/test.pcap -o lo
>> sudo trafgen/trafgen -i /tmp/test.pcap -o lo -b 1pps
>>
>> After this change will be applied I think I can try to add option to
>> netsniff-ng tool to merging pcap files.
>>
>> Regards,
>> Vadim Kochan
>
> Hm,
>
> I just realized that netsniff-ng already has pcap -> device sending
> ability, but at max rate.
> But I am stil think that it is good to have it in trafgen. Lets see if
> Tobias/Daniel will agree or not on it ...

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [netsniff-ng] Few Questions I could not find answer for :)

2016-12-04 Thread Vadim Kochan
On Sun, Dec 4, 2016 at 2:09 PM, Vadim Kochan <vadi...@gmail.com> wrote:
> On Tue, Nov 22, 2016 at 10:32 AM, Shaked, Asaf <asaf.sha...@verint.com> wrote:
>> Cool... good to see we're on the same page... thanks for the prompt response 
>> :)
>>
>> Asaf.
>>
>> -Original Message-
>> From: Vadim Kochan [mailto:vadi...@gmail.com]
>> Sent: יום ג 22 נובמבר 2016 02:00
>> To: Shaked, Asaf
>> Cc: netsniff-ng@googlegroups.com
>> Subject: Re: [netsniff-ng] Few Questions I could not find answer for :)
>>
>> On Mon, Nov 21, 2016 at 03:37:48PM +, Shaked, Asaf wrote:
>> Hi Asaf,
>>
>>> Hi,
>>>
>>> Just love this toolkit :)
>>>
>>> Few questions, if you may...
>>>
>>> 1)  Is it possible to merge pcaps using the toolkit ?
>> As fas as I know - no, but I think it would be good to have such feature via 
>> such example:
>>
>> netsniff-ng -i {in_dir} -o {merged_file}.pcap
>>
>>>
>>> 2)  Is it possible to replay a list of pcaps consecutively ?
>>>
>> As I know, it is posible only to redirect packets from one device to 
>> another, but I really thing that it might be added to trafgen.
>>
>>> 3)  Is it possible to control pcaps replay bitrate ? (aiming for 
>>> original capture average Bitrate, rather than "as fast as possible")
>>>
>> So if option #2 will be implemented then I think it wold be easy have it in 
>> trafgen as there ia already rate limit option.
>>
>>> Thanks,
>>>
>>> Asaf.
>>>
>>
>> Regards,
>> Vadim Kochan
>>
>
> Hi Asaf,
>
> I tried to extend trafgen with ability to send packets from pcap file,
> and you can try it if you able to fetch & build trafgen from:
>
> https://github.com/vkochan/netsniff-ng/tree/trafgen_pcap_send
>
> By default original timestamps are considered when sending traffic
> from pcap, but you
> can reset sending rate via -b/-t options:
>
> sudo trafgen/trafgen -i /tmp/test.pcap -o lo
> sudo trafgen/trafgen -i /tmp/test.pcap -o lo -b 1pps
>
> After this change will be applied I think I can try to add option to
> netsniff-ng tool to merging pcap files.
>
> Regards,
> Vadim Kochan

Hm,

I just realized that netsniff-ng already has pcap -> device sending
ability, but at max rate.
But I am stil think that it is good to have it in trafgen. Lets see if
Tobias/Daniel will agree or not on it ...

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [netsniff-ng] Few Questions I could not find answer for :)

2016-12-04 Thread Vadim Kochan
On Tue, Nov 22, 2016 at 10:32 AM, Shaked, Asaf <asaf.sha...@verint.com> wrote:
> Cool... good to see we're on the same page... thanks for the prompt response 
> :)
>
> Asaf.
>
> -Original Message-
> From: Vadim Kochan [mailto:vadi...@gmail.com]
> Sent: יום ג 22 נובמבר 2016 02:00
> To: Shaked, Asaf
> Cc: netsniff-ng@googlegroups.com
> Subject: Re: [netsniff-ng] Few Questions I could not find answer for :)
>
> On Mon, Nov 21, 2016 at 03:37:48PM +, Shaked, Asaf wrote:
> Hi Asaf,
>
>> Hi,
>>
>> Just love this toolkit :)
>>
>> Few questions, if you may...
>>
>> 1)  Is it possible to merge pcaps using the toolkit ?
> As fas as I know - no, but I think it would be good to have such feature via 
> such example:
>
> netsniff-ng -i {in_dir} -o {merged_file}.pcap
>
>>
>> 2)  Is it possible to replay a list of pcaps consecutively ?
>>
> As I know, it is posible only to redirect packets from one device to another, 
> but I really thing that it might be added to trafgen.
>
>> 3)  Is it possible to control pcaps replay bitrate ? (aiming for 
>> original capture average Bitrate, rather than "as fast as possible")
>>
> So if option #2 will be implemented then I think it wold be easy have it in 
> trafgen as there ia already rate limit option.
>
>> Thanks,
>>
>> Asaf.
>>
>
> Regards,
> Vadim Kochan
>

Hi Asaf,

I tried to extend trafgen with ability to send packets from pcap file,
and you can try it if you able to fetch & build trafgen from:

https://github.com/vkochan/netsniff-ng/tree/trafgen_pcap_send

By default original timestamps are considered when sending traffic
from pcap, but you
can reset sending rate via -b/-t options:

sudo trafgen/trafgen -i /tmp/test.pcap -o lo
sudo trafgen/trafgen -i /tmp/test.pcap -o lo -b 1pps

After this change will be applied I think I can try to add option to
netsniff-ng tool to merging pcap files.

Regards,
Vadim Kochan

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


RE: [netsniff-ng] Few Questions I could not find answer for :)

2016-11-22 Thread Shaked, Asaf
Cool... good to see we're on the same page... thanks for the prompt response :)

Asaf.

-Original Message-
From: Vadim Kochan [mailto:vadi...@gmail.com] 
Sent: יום ג 22 נובמבר 2016 02:00
To: Shaked, Asaf
Cc: netsniff-ng@googlegroups.com
Subject: Re: [netsniff-ng] Few Questions I could not find answer for :)

On Mon, Nov 21, 2016 at 03:37:48PM +, Shaked, Asaf wrote:
Hi Asaf,

> Hi,
> 
> Just love this toolkit :)
> 
> Few questions, if you may...
> 
> 1)  Is it possible to merge pcaps using the toolkit ?
As fas as I know - no, but I think it would be good to have such feature via 
such example:

netsniff-ng -i {in_dir} -o {merged_file}.pcap

> 
> 2)  Is it possible to replay a list of pcaps consecutively ?
> 
As I know, it is posible only to redirect packets from one device to another, 
but I really thing that it might be added to trafgen.

> 3)  Is it possible to control pcaps replay bitrate ? (aiming for original 
> capture average Bitrate, rather than "as fast as possible")
> 
So if option #2 will be implemented then I think it wold be easy have it in 
trafgen as there ia already rate limit option.

> Thanks,
> 
> Asaf.
> 

Regards,
Vadim Kochan

-- 
This electronic message may contain proprietary and confidential 
information of Verint Systems Inc., its affiliates and/or subsidiaries.
The information is intended to be for the use of the individual(s) or
entity(ies) named above. If you are not the intended recipient (or 
authorized to receive this e-mail for the intended recipient), you may not 
use, copy, disclose or distribute to anyone this message or any information 
contained in this message. If you have received this electronic message in 
error, please notify us by replying to this e-mail.

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [netsniff-ng] Few Questions I could not find answer for :)

2016-11-21 Thread Vadim Kochan
On Mon, Nov 21, 2016 at 03:37:48PM +, Shaked, Asaf wrote:
Hi Asaf,

> Hi,
> 
> Just love this toolkit :)
> 
> Few questions, if you may...
> 
> 1)  Is it possible to merge pcaps using the toolkit ?
As fas as I know - no, but I think it would be good to have such feature
via such example:

netsniff-ng -i {in_dir} -o {merged_file}.pcap

> 
> 2)  Is it possible to replay a list of pcaps consecutively ?
> 
As I know, it is posible only to redirect packets from one device to
another, but I really thing that it might be added to trafgen.

> 3)  Is it possible to control pcaps replay bitrate ? (aiming for original 
> capture average Bitrate, rather than "as fast as possible")
> 
So if option #2 will be implemented then I think it wold be easy have it
in trafgen as there ia already rate limit option.

> Thanks,
> 
> Asaf.
> 

Regards,
Vadim Kochan

-- 
You received this message because you are subscribed to the Google Groups 
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to netsniff-ng+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.