FWIW, it's not that you have a large pcap. That's not really the issue here. The problem is that you have a set of IP addresses and the randomization feature doesn't guarantee a 1 to 1 mapping. Granted, statistically, that's more likely to happen with a larger pcap full of more hosts talking, but that's not the root cause.
Anyways, I opened a ticket to track this issue here: http://tcpreplay.synfin.net/ticket/525 Glad you were able to find a work around! Regards, Aaron On Wed, Sep 26, 2012 at 9:32 PM, Eric Formo <e...@vineyardnetworks.com> wrote: > Yes, the traffic is from internal networks, and by chance both the > client/server is on the same subnet. Luckily, I just had to do a few > initial modifications moving a couple of my flows from one ip schema to > the same one as the rest, but once I got all the flows on the same subnet > the --pnat option was a great workaround. > > It would be great to have a future enhancement for tcprewrite to be able > to work with 2GB+ files and the --seed option. I think with a lot of > networking company's striving towards the 10Gbps connection speeds, > working with larger pcap's will become more prevalent. > > Thanks for the help! > > -----Original Message----- > From: Aaron Turner [mailto:synfina...@gmail.com] > Sent: September-25-12 5:14 PM > To: Main forum for tcpreplay > Subject: Re: [Tcpreplay-users] tcprewrite seed issues > > Ok. Curious... is the traffic in this pcap from an internal > network(s) or public internet? If the former, you could just use the > --pnat option to rewrite the traffic to a new /8 which would allow you to > create 255 unique copies. Obviously though, if the traffic is from say a > public webserver, that won't work. > > If not, i can probably hook you up with a bit of custom code which I think > will work. As I said, it won't be random, but it should do a better job > of creating unique flows.... i think. kinda depends on the data in the > pcap. :) > > On Wed, Sep 26, 2012 at 1:03 AM, Eric Formo <e...@vineyardnetworks.com> > wrote: >> Basically, my current project is to take a 1-3GB file, rewrite and >> copy it >> 20 times in order to do some performance testing. I just need to make >> sure they are different in each of the 20 copies, but the packet >> tuples stay the same or else my test breaks. >> >> -----Original Message----- >> From: Aaron Turner [mailto:synfina...@gmail.com] >> Sent: September-25-12 3:31 PM >> To: Main forum for tcpreplay >> Subject: Re: [Tcpreplay-users] tcprewrite seed issues >> >> Unfortunately, there is nothing to say that a single flow won't span >> two different seeds, which would thing break the flow. >> >> What are you exact needs anyways? Do they really need to be random or >> are you just trying to get different values then what is stored? >> >> There are some quick code changes I could do as a one off which would >> give you "different, but unique" new values, but they wouldn't be >> random at all. >> >> >> On Tue, Sep 25, 2012 at 11:05 PM, Eric Formo >> <e...@vineyardnetworks.com> >> wrote: >>> That is what I suspected. I think my next best bet would be to split >>> them, check to make sure the connections are the same, then rewrite >>> however many times I need, then merge them back together. Could a >>> solution could be to have multiple sets of seeds? 'seed A' for 2 >>> seconds, 'seed B' for 2 seconds, etc... >>> Thanks for the quick response though, let me know if anyone has a >>> better suggestion for a workaround than what mine is above. >>> >>> -----Original Message----- >>> From: Aaron Turner [mailto:synfina...@gmail.com] >>> Sent: September-25-12 3:00 PM >>> To: Main forum for tcpreplay >>> Subject: Re: [Tcpreplay-users] tcprewrite seed issues >>> >>> I can guess. The randomization feature isn't that smart... just a >>> simple algorithm and doesn't check for uniqueness. I never bothered >>> to check that it was a good implementation for large data sets. >>> >>> http://tcpreplay.synfin.net/browser/branches/3.4/src/tcpedit/edit_pac >>> k >>> et.c >>> #L125 >>> >>> If you have enough packets with enough IP addresses, sooner or later >>> you'll end up with a collision (IP1 and IP2 => NewIP). I guess I >>> could come up with a better algorithm which avoids duplicate results. >>> Probably be a while before that happens though. >>> >>> >>> >>> On Tue, Sep 25, 2012 at 10:04 PM, Eric Formo >>> <e...@vineyardnetworks.com> >>> wrote: >>>> Using tcprewrite with the seed argument is not working for me right >>>> now. I am rewriting 2.5G pcap that is about 7 seconds long with >>>> tcprewrite --seed=$(($RANDOM*$RANDOM)). After rewriting, my pcap >>>> goes from having >>>> 75835 flows down to 43811 because tcprewrite seems to be giving >>>> different IP address's the same IP which put's some of the >>>> connections >>> in the same tuple. >>>> >>>> I picked out what should be 2 small flows for an example of what is >>>> happening throughout the rewrite multiple times. Looking at packet >>>> 4, you can see that rewrite has given the packet the same IP as the >>>> first >>>> 3 packets. Is there a reason this is happening? I tried using the >>>> same command on the original_sample.pcap and the rewrite worked >>>> perfectly. So the problem only exists when the file is 2.5 GB. >>>> >>>> >>>> >>>> My original_sample.pcap: >>>> 1 0.000000 103.0.2.140 -> 103.0.6.140 TCP edtools > http [SYN] >> Seq=0 >>>> Win=2048 Len=0 WS=9 >>>> >>>> 2 0.005000 103.0.6.140 -> 103.0.2.140 TCP http > edtools [SYN, >>> ACK] >>>> Seq=0 Ack=1 Win=2048 Len=0 WS=9 >>>> >>>> 3 0.023000 103.0.2.140 -> 103.0.6.140 TCP edtools > http [ACK] >>> Seq=1 >>>> Ack=1 Win=1048576 Len=0 >>>> >>>> 4 0.174000 103.0.2.143 -> 103.0.6.143 TCP edtools > http [SYN] >>> Seq=0 >>>> Win=2048 Len=0 WS=9 >>>> >>>> 5 0.179000 103.0.6.143 -> 103.0.2.143 TCP http > edtools [SYN, >>> ACK] >>>> Seq=0 Ack=1 Win=2048 Len=0 WS=9 >>>> >>>> 6 0.208000 103.0.2.143 -> 103.0.6.143 TCP edtools > http [ACK] >>> Seq=1 >>>> Ack=1 Win=1048576 Len=0 >>>> >>>> 7 0.403000 103.0.2.140 -> 103.0.6.140 HTTP GET >>>> /services/auth/?api_key=610346c7619bc6bd51afbbd2739f8d1f&api_sig=b99 >>>> 3 >>>> d >>>> e5486706c18a08757948c04963e&perms=write&frob=72157627531937949-7b4cf >>>> 5 >>>> 0 >>>> cc91b7232-326096 >>>> HTTP/1.1 >>>> >>>> 8 0.475000 103.0.6.140 -> 103.0.2.140 TCP http > edtools [ACK] >>> Seq=1 >>>> Ack=500 Win=1048576 Len=0 >>>> >>>> 9 0.522000 103.0.2.143 -> 103.0.6.143 HTTP GET >>>> /services/auth/?api_key=610346c7619bc6bd51afbbd2739f8d1f&api_sig=b99 >>>> 3 >>>> d >>>> e5486706c18a08757948c04963e&perms=write&frob=72157627531937949-7b4cf >>>> 5 >>>> 0 >>>> cc91b7232-326096 >>>> HTTP/1.1 >>>> >>>> 10 0.528000 103.0.6.143 -> 103.0.2.143 TCP http > edtools [ACK] >>> Seq=1 >>>> Ack=500 Win=1048576 Len=0 >>>> >>>> >>>> >>>> The rewrite_sample: >>>> 1 0.000000 39.66.178.173 -> 39.66.182.173 TCP edtools > http [SYN] >>> Seq=0 >>>> Win=2048 Len=0 WS=9 >>>> >>>> 2 0.005000 39.66.182.173 -> 39.66.178.173 TCP http > edtools [SYN, >>> ACK] >>>> Seq=0 Ack=1 Win=2048 Len=0 WS=9 >>>> >>>> 3 0.023000 39.66.178.173 -> 39.66.182.173 TCP edtools > http [ACK] >>> Seq=1 >>>> Ack=1 Win=1048576 Len=0 >>>> >>>> 4 0.174000 39.66.178.173 -> 39.66.182.173 TCP [TCP Port numbers >>> reused] >>>> edtools > http [SYN] Seq=0 Win=2048 Len=0 WS=9 >>>> >>>> 5 0.179000 39.66.182.173 -> 39.66.178.173 TCP http > edtools [SYN, >>> ACK] >>>> Seq=0 Ack=1 Win=2048 Len=0 WS=9 >>>> >>>> 6 0.208000 39.66.178.173 -> 39.66.182.173 TCP edtools > http [ACK] >>> Seq=1 >>>> Ack=1 Win=1048576 Len=0 >>>> >>>> 7 0.403000 39.66.178.173 -> 39.66.182.173 HTTP [TCP ACKed lost >>> segment] >>>> [TCP Retransmission] GET >>>> /services/auth/?api_key=610346c7619bc6bd51afbbd2739f8d1f&api_sig=b99 >>>> 3 >>>> d >>>> e5486706c18a08757948c04963e&perms=write&frob=72157627531937949-7b4cf >>>> 5 >>>> 0 >>>> cc91b7232-326096 >>>> HTTP/1.1 >>>> >>>> 8 0.475000 39.66.182.173 -> 39.66.178.173 TCP http > edtools [ACK] >>>> Seq=285549713 Ack=2574765636 Win=1048576 Len=0 >>>> >>>> 9 0.522000 39.66.178.173 -> 39.66.182.173 HTTP GET >>>> /services/auth/?api_key=610346c7619bc6bd51afbbd2739f8d1f&api_sig=b99 >>>> 3 >>>> d >>>> e5486706c18a08757948c04963e&perms=write&frob=72157627531937949-7b4cf >>>> 5 >>>> 0 >>>> cc91b7232-326096 >>>> HTTP/1.1 >>>> >>>> 10 0.528000 39.66.182.173 -> 39.66.178.173 TCP http > edtools [ACK] >>> Seq=1 >>>> Ack=500 Win=1048576 Len=0 >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Eric >>>> >>>> >>>> >>>> >>>> -------------------------------------------------------------------- >>>> - >>>> - >>>> -------- >>>> Live Security Virtual Conference >>>> Exclusive live event will cover all the ways today's security and >>>> threat landscape has changed and how IT managers can respond. >>>> Discussions will include endpoint security, mobile security and the >>>> latest in malware threats. >>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>> _______________________________________________ >>>> Tcpreplay-users mailing list >>>> Tcpreplay-users@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/tcpreplay-users >>>> Support Information: http://tcpreplay.synfin.net/trac/wiki/Support >>> >>> >>> >>> -- >>> Aaron Turner >>> http://synfin.net/ Twitter: @synfinatic >>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix >>> & Windows Those who would give up essential Liberty, to purchase a >>> little temporary Safety, deserve neither Liberty nor Safety. >>> -- Benjamin Franklin >>> "carpe diem quam minimum credula postero" >>> >>> --------------------------------------------------------------------- >>> - >>> ---- >>> ---- >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. >>> Discussions will include endpoint security, mobile security and the >>> latest in malware threats. >>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> _______________________________________________ >>> Tcpreplay-users mailing list >>> Tcpreplay-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/tcpreplay-users >>> Support Information: http://tcpreplay.synfin.net/trac/wiki/Support >>> >>> --------------------------------------------------------------------- >>> - >>> -------- >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. >>> Discussions will include endpoint security, mobile security and the >>> latest in malware threats. >>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> _______________________________________________ >>> Tcpreplay-users mailing list >>> Tcpreplay-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/tcpreplay-users >>> Support Information: http://tcpreplay.synfin.net/trac/wiki/Support >> >> >> >> -- >> Aaron Turner >> http://synfin.net/ Twitter: @synfinatic >> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix >> & Windows Those who would give up essential Liberty, to purchase a >> little temporary Safety, deserve neither Liberty nor Safety. >> -- Benjamin Franklin >> "carpe diem quam minimum credula postero" >> >> ---------------------------------------------------------------------- >> ---- >> ---- >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. >> Discussions will include endpoint security, mobile security and the >> latest in malware threats. >> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Tcpreplay-users mailing list >> Tcpreplay-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/tcpreplay-users >> Support Information: http://tcpreplay.synfin.net/trac/wiki/Support >> >> ---------------------------------------------------------------------- >> -------- >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. >> Discussions will include endpoint security, mobile security and the >> latest in malware threats. >> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Tcpreplay-users mailing list >> Tcpreplay-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/tcpreplay-users >> Support Information: http://tcpreplay.synfin.net/trac/wiki/Support > > > > -- > Aaron Turner > http://synfin.net/ Twitter: @synfinatic > http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & > Windows Those who would give up essential Liberty, to purchase a little > temporary Safety, deserve neither Liberty nor Safety. > -- Benjamin Franklin > "carpe diem quam minimum credula postero" > > -------------------------------------------------------------------------- > ---- > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and threat > landscape has changed and how IT managers can respond. Discussions will > include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Tcpreplay-users mailing list > Tcpreplay-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/tcpreplay-users > Support Information: http://tcpreplay.synfin.net/trac/wiki/Support > > ------------------------------------------------------------------------------ > How fast is your code? > 3 out of 4 devs don\\\'t know how their code performs in production. > Find out how slow your code is with AppDynamics Lite. > http://ad.doubleclick.net/clk;262219672;13503038;z? > http://info.appdynamics.com/FreeJavaPerformanceDownload.html > _______________________________________________ > Tcpreplay-users mailing list > Tcpreplay-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/tcpreplay-users > Support Information: http://tcpreplay.synfin.net/trac/wiki/Support -- Aaron Turner http://synfin.net/ Twitter: @synfinatic http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. -- Benjamin Franklin "carpe diem quam minimum credula postero" ------------------------------------------------------------------------------ How fast is your code? 3 out of 4 devs don\\\'t know how their code performs in production. Find out how slow your code is with AppDynamics Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? http://info.appdynamics.com/FreeJavaPerformanceDownload.html _______________________________________________ Tcpreplay-users mailing list Tcpreplay-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tcpreplay-users Support Information: http://tcpreplay.synfin.net/trac/wiki/Support
