The input is consistent in that it all has stx at the beginning of each 'event.' I'm leaning towards regex. When you say:
" find stx, stuff lines until I see the next stx, then dump and continue" Might I trouble you for an example of how you do that? I can find stx, I can find etx using something along the lines of : a = [m.start() for m in re.finditer(r"<devicename>", line)] but then I get a little lost, mostly because I have some lines that have "data data [\x03][\x02] data" and then to the next line. More succinctly, the stx aren't always at the beginning of the line, etx not always at the end. No problem, I can find them, but then I'm guessing I would have to write to a buffer starting with stx, keep writing to the buffer until I get to etx, write the buffer to file (or send it over the socket, either way is fine) then continue on. The fact that 'events' span multiple lines is challenging me. On Tue, Dec 27, 2016 at 3:55 PM, David Rock <[email protected]> wrote: > * richard kappler <[email protected]> [2016-12-27 15:39]: > > I was actually working somewhat in that direction while I waited. I had > in > > mind to use something along the lines of: > > > > > > stx = '\x02' > > etx = '\x03' > > line1 = "" > > > > with open('original.log', 'r') as f1: > > with open('new.log', 'w') as f2: > > for line in f1: > > if stx in line: > > line1 = line1 + line > > if not stx in line: > > if not etx in line: > > line1 = line1 + line > > if etx in line: > > line1 = line1 + line + '\n' > > f2.write(line1) > > line1 = "" > > > > > > but that didn't work. It neither broke each line on etx (multiple events > > with stx and etx on one line) nor did it concatenate the multi-line > events. > > A big part of the challenge sounds like it's inconsistent data > formatting. You are going to have to identify some way to reliably > check for the beginning/end of your data for it to work. Do you know if > you will always have \x02 at the start of a section of input, for example? > > The way I usually do log parsing in that case is use the stx as a flag > to start doing other things (ie, if I find stx, stuff lines until I see > the next stx, then dump and continue). If you have intermediary data > that is not between your stx and etx (comment lines, other data that you > don't want), then it gets a lot harder. > > If you don't have at least a marginally consistent input, your only real > option is probably going to be scanning by character and looking for the > \x02 and \x03 to get a glob of data, then parse that glob with some kind > of xml parser, since the data between those two is likely safe-ish. > > -- > David Rock > [email protected] > _______________________________________________ > Tutor maillist - [email protected] > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > _______________________________________________ Tutor maillist - [email protected] To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
