I have something working for me now. If will probably be a couple of weeks before I have time to do much more with it, so posting it now in case anyone finds it useful or wants to improve it.
It has only been tested with IxCatapult log files so far, but hopefully it will work with any well-behaved file type supported by wiretap. It is pretty inefficient (linear search for place to insert, inserting from wrong end, long queue length to suit the file I've been testing with, ....), but for now I don't mind waiting for a few seconds to get a nicely ordered log. Regards, Martin On Mon, Sep 17, 2012 at 11:02 AM, Martin Mathieson < [email protected]> wrote: > I think I will write a separate wiretap console app (tonight, if I get > carried away). > > My log files tend to be long (100s MB), so I don't think a script such as > yours would work well for me. > > Ultimately it would be good to have this functionality in (probably) > editcap, and if there is demand, make it another option when exporting > frames from wireshark. > > Regards, > Martin > > > > On Mon, Sep 17, 2012 at 3:15 PM, James Howard Young <[email protected]>wrote: > >> Hello Martin, >> >> I've used mergecap to do this. >> >> It's a bit of a hacky solution but I used to split the >> original out-or-order packets out to separate 1 packet >> trace files. I then used mergecap's default chronological >> merging behavior to piece the sections back into a single >> file. The problem with this approach is that mergecap can >> ONLY work with about 512 trace files at a time. Depending >> on how many out of order sections the trace file contains >> this can get very tedious to do manually. >> >> I eventually cooked up a shell script that helped automate >> the process. But the script was very slow, but it did >> (eventually) get the work done. >> >> The script's main loop would fix-up one negative delta >> section at a time. It used capinfos -o report to determine >> if the trace file had any out-of-order packets. If so it >> ran a tshark -td report grepping for any negative delta >> time records and piping it to head -1. It would extract >> the frame number of the 1st negative delta time packet >> and use that particular frame number to split the trace >> file into two parts. The first part would contain all the >> frames up to the out-of-order frame. The second part >> would contain all frame from the out-of-order frame onward. >> The two parts would be merged back together with mergecap. >> Then the newly reconstituted trace file would be processed >> again and again until the capinfos -o report indicated >> that the was in strict chronological order. >> >> Unfortunately I lost that particular script but it shouldn't >> be too hard to re-create. >> >> But I think augmenting editcap or mergecap or even adding >> a new wiretap based console app to do this would be useful. >> I've thought about this before but I simply have NOT had the >> time to invest in following up. I suspect the ability of >> wireshark with pcapng to support multiple concurrent inputs >> might make it more likely for to have trace files with >> "out-of-order" packets. >> >> I hope this helps, >> >> Jim Y. >> >> >> On 9/17/12 9:21 AM, "Martin Mathieson" <[email protected] >> > >> wrote: >> >> >Hi, >> > >> > >> >I work with log files that are created from multiple sources, and >> >although the timestamps are in good synchronisation, they are often >> >written to the log file slightly out of order (up to a few milliseconds). >> > >> > >> >editcap -S supports rewriting the timestamps to be in strict >> >chronological order (assuming the frame order will be correct), but this >> >is the opposite of what I need. >> > >> > >> >So, I'm considering adding a new flag to editcap, or maybe creating a >> >separate console program using wiretap (I haven't thought through how >> >difficult it might be to support other options at the same time). It >> >would work something like this: >> >- specify a number N (say 200), that would serve as the number of frames >> >in the ordering buffer >> >- when a new frame is read from the input file, insert it into its >> >correct time order within the (up-to) N frames in the buffer. Hopefully >> >not a linear search :) >> >- write the earliest frame to the output file >> >- when we reach the end of the input file, just flush the sorted buffer >> >to the output file >> > >> > >> >Not sure if I'd keep the buffered frames in memory, or go back and reread >> >them when it was time to write them out. >> > >> > >> > >> > >> >Am I missing an existing way to get this functionality? Am I the only >> >person who needs this functionality? >> > >> > >> >Thanks, >> >Martin >> > >> >> >> >> >> ___________________________________________________________________________ >> Sent via: Wireshark-dev mailing list <[email protected]> >> Archives: http://www.wireshark.org/lists/wireshark-dev >> Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev >> mailto:[email protected] >> ?subject=unsubscribe >> > >
/* Reorder the frames from an input dump file, and write to output dump file. * * $Id: reorder.c 42553 2012-05-10 16:10:44Z martinm $ * * Wireshark - Network traffic analyzer * By Gerald Combs <[email protected]> * Copyright 1998 Gerald Combs * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License along * with this program; if not, write to the Free Software Foundation, Inc., * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. * */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <stdarg.h> #include "wtap.h" /* #define REORDER_DEBUG */ /* Show command-line usage */ /* TODO: add reoder list length as an optional param? */ static void usage(void) { printf("usage: reorder <infile> <outfile>\n"); } /* Remember where this frame was in the file */ typedef struct FrameRecord_t { gint64 offset; guint32 length; struct wtap_nstime time; /* List item pointers */ struct FrameRecord_t *prev; struct FrameRecord_t *next; } FrameRecord_t; /* This is pretty big, but I don't mind waiting a few seconds */ #define MAX_REORDER_LIST_LENGTH 3000 static int g_FrameRecordCount; static FrameRecord_t *g_FrameListHead; /* Is time1 earlier than time2? */ static gboolean isEarlierTime(struct wtap_nstime time1, struct wtap_nstime time2) { if (time1.secs < time2.secs) { return TRUE; } if (time1.secs == time2.secs) { return (time1.nsecs < time2.nsecs); } else { return FALSE; } } /* Is the reorder list empty? */ static gboolean ReorderListEmpty(void) { return (g_FrameRecordCount == 0); } /* Is the reorder list full? */ static gboolean ReorderListFull(void) { return (g_FrameRecordCount >= MAX_REORDER_LIST_LENGTH); } /* Add a new frame to the reorder list */ /* TODO: would presumably be much faster to add it from the end!!! */ static void ReorderListAdd(gint64 offset, guint32 length, struct wtap_nstime time) { FrameRecord_t *tmp; FrameRecord_t *newFrameRecord = g_malloc(sizeof(FrameRecord_t)); /* Populate fields */ #ifdef REORDER_DEBUG printf("\nAdded with offset=%06llu, length=%05u, secs=%lu, nsecs=%d\n", offset, length, time.secs, time.nsecs); #endif newFrameRecord->offset = offset; newFrameRecord->length = length; newFrameRecord->time = time; /* We will definitely add it below, so inc counter */ g_FrameRecordCount++; /* First time, this will be the head */ if (g_FrameListHead == NULL) { #ifdef REORDER_DEBUG printf("this item will be head - only item\n"); #endif g_FrameListHead = newFrameRecord; newFrameRecord->prev = NULL; newFrameRecord->next = NULL; return; } /* Look for the place in the list where this item fits */ tmp = g_FrameListHead; while (tmp != NULL) { if (isEarlierTime(time, tmp->time)) { #ifdef REORDER_DEBUG printf("Time was less, writing before element\n"); #endif /* Insert newFrameRecord before tmp */ /* Fix up previous item */ if (tmp != g_FrameListHead) { newFrameRecord->prev = tmp->prev; newFrameRecord->prev->next = newFrameRecord; } else { g_FrameListHead = newFrameRecord; } /* Fix up next item */ newFrameRecord->next = tmp; tmp->prev = newFrameRecord; return; } /* Didn't find an item to insert in front of */ if (tmp->next == NULL) { #ifdef REORDER_DEBUG printf("Reached the end of the list, so insert here\n"); #endif /* We are the new last item */ tmp->next = newFrameRecord; newFrameRecord->prev = tmp; newFrameRecord->next = NULL; return; } else { /* Move onto the next item */ tmp = tmp->next; } } } /* Dump the first item in the reorder list to the output file, and pop it */ static void ReorderListDumpFirst(wtap *wth, wtap_dumper *pdh) { union wtap_pseudo_header pseudo_header; int err; gchar *errinfo; const struct wtap_pkthdr *phdr; guint8 buf[8000]; struct wtap_pkthdr new_phdr; FrameRecord_t *prev_head = g_FrameListHead; #ifdef REORDER_DEBUG printf("\nDumping frame (offset=%llu, length=%u) (%u items in list)\n", g_FrameListHead->offset, g_FrameListHead->length, g_FrameRecordCount); #endif /* Re-read the first frame from the stored location */ wtap_seek_read(wth, g_FrameListHead->offset, &pseudo_header, buf, g_FrameListHead->length, &err, &errinfo); #ifdef REORDER_DEBUG printf("re-read: err is %u, buf is (%s)\n", err, buf); #endif /* Get packet header */ phdr = wtap_phdr(wth); #ifdef REORDER_DEBUG printf("re-read: secs are %lu, nsecs are %u\n", phdr->ts.secs, phdr->ts.nsecs); #endif /* Copy, and set length. TODO: why does length need to be set? */ memcpy((void*)&new_phdr, phdr, sizeof(struct wtap_pkthdr)); new_phdr.len = g_FrameListHead->length; /* Dump frame to outfile */ if (!wtap_dump(pdh, &new_phdr, &pseudo_header, buf, &err)) { #ifdef REORDER_DEBUG printf("Error (%s) writing frame to outfile\n", wtap_strerror(err)); #endif exit(1); } /* Now remove the first item from the list */ if (g_FrameListHead->next == NULL) { g_FrameListHead = NULL; } else { g_FrameListHead->next->prev = NULL; g_FrameListHead = g_FrameListHead->next; } /* And free the struct */ g_free(prev_head); g_FrameRecordCount--; #ifdef REORDER_DEBUG printf("Frame written, %u remaining\n", g_FrameRecordCount); #endif } /********************************************************************/ /* Main function. */ /********************************************************************/ int main(int argc, char *argv[]) { wtap *wth = NULL; wtap_dumper *pdh = NULL; int err; gchar *err_info; gint64 data_offset; const struct wtap_pkthdr *phdr; guint32 read_count = 0; /* 1st arg is infile, 2nd arg is outfile */ char *infile; char *outfile; if (argc == 3) { infile = argv[1]; outfile = argv[2]; } else { usage(); exit(1); } /* Open infile */ wth = wtap_open_offline(infile, &err, &err_info, TRUE); if (wth == NULL) { printf("reorder: Can't open %s: %s\n", infile, wtap_strerror(err)); exit(1); } #ifdef REORDER_DEBUG printf("file_type is %u\n", wtap_file_type(wth)); #endif /* Open outfile (same filetype/encap as input file) */ pdh = wtap_dump_open(outfile, wtap_file_type(wth), wtap_file_encap(wth), 65535, FALSE, &err); if (pdh == NULL) { printf("Failed to open output file: (%s) - error %s\n", outfile, wtap_strerror(err)); exit(1); } /* Read each frame from infile */ while (wtap_read(wth, &err, &err_info, &data_offset)) { read_count++; phdr = wtap_phdr(wth); /* Add it to the reordering list */ ReorderListAdd(data_offset, phdr->len, phdr->ts); /* If/when the list gets full, dump the first item out */ if (ReorderListFull()) { #ifdef REORDER_DEBUG printf("List is full, dumping first!\n"); #endif /* Write out the first one */ ReorderListDumpFirst(wth, pdh); } } /* Flush out the remaining (ordered) frames */ while (!ReorderListEmpty()) { ReorderListDumpFirst(wth, pdh); } /* Close outfile */ if (!wtap_dump_close(pdh, &err)) { printf("reorder: Error closing %s: %s\n", outfile, wtap_strerror(err)); exit(1); } /* TODO: write how many frames, and how many were out of order? */ /* Finally, close infile */ wtap_fdclose(wth); return 0; }
___________________________________________________________________________ Sent via: Wireshark-dev mailing list <[email protected]> Archives: http://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev mailto:[email protected]?subject=unsubscribe
