Multiline regex
I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; I'm wanting to grab the information out in chunks, so createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; would be what my regex would grab. I'm currently only able to grab out the first line and part of the second line, but no more. regex is as follows my_regexp = re.compile(createNode\ animCurve.*\n[\t*setAttr.*\n]*) I've run several variations of this, but none return me all of the expected information. Is there something special that needs to be done to have the regexp grab any number of the setAttr lines without specification? Brandon L. Harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
Slurp the entire file into a string and pick out the fields you need. Sent from my iPhone 4. On Jul 21, 2010, at 10:42 AM, Brandon Harris brandon.har...@reelfx.com wrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; I'm wanting to grab the information out in chunks, so createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; would be what my regex would grab. I'm currently only able to grab out the first line and part of the second line, but no more. regex is as follows my_regexp = re.compile(createNode\ animCurve.*\n[\t*setAttr.*\n]*) I've run several variations of this, but none return me all of the expected information. Is there something special that needs to be done to have the regexp grab any number of the setAttr lines without specification? Brandon L. Harris -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
what do you mean by slurp the entire file? I'm trying to use regular expressions because line by line parsing will be too slow. And example file would have somewhere in the realm of 6 million lines of code. Brandon L. Harris Rodrick Brown wrote: Slurp the entire file into a string and pick out the fields you need. Sent from my iPhone 4. On Jul 21, 2010, at 10:42 AM, Brandon Harris brandon.har...@reelfx.com wrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; I'm wanting to grab the information out in chunks, so createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; would be what my regex would grab. I'm currently only able to grab out the first line and part of the second line, but no more. regex is as follows my_regexp = re.compile(createNode\ animCurve.*\n[\t*setAttr.*\n]*) I've run several variations of this, but none return me all of the expected information. Is there something special that needs to be done to have the regexp grab any number of the setAttr lines without specification? Brandon L. Harris -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
On Wed, Jul 21, 2010 at 8:12 PM, Brandon Harris brandon.har...@reelfx.comwrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Do you have to use only regex? If not, I'd certainly suggest 'pyparsing'. It's a pleasure to use and very easy on the eye too, if you know what I mean. I'm wanting to grab the information out in chunks, so -- Eknath Venkataramani -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
At the moment I'm trying to stick with built in python modules to create tools for a much larger pipeline on multiple OSes. Brandon L. Harris Eknath Venkataramani wrote: On Wed, Jul 21, 2010 at 8:12 PM, Brandon Harris brandon.har...@reelfx.com mailto:brandon.har...@reelfx.com wrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Do you have to use only regex? If not, I'd certainly suggest 'pyparsing'. It's a pleasure to use and very easy on the eye too, if you know what I mean. I'm wanting to grab the information out in chunks, so -- Eknath Venkataramani -- http://mail.python.org/mailman/listinfo/python-list
RE: Multiline regex
I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; I'm wanting to grab the information out in chunks, so createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; would be what my regex would grab. I'm currently only able to grab out the first line and part of the second line, but no more. regex is as follows my_regexp = re.compile(createNode\ animCurve.*\n[\t*setAttr.*\n]*) I've run several variations of this, but none return me all of the expected information. Is there something special that needs to be done to have the regexp grab any number of the setAttr lines without specification? Brandon L. Harris Aren't you making life too complicated for yourself? blocks = [] for line in yourFile: if line.startswith(createNode): if currentBlock: blocks.append(currentBlock) currentBlock = [line] else: currentBlock.append(line) blocks.append(currentBlock) Cheers, Drea -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
Brandon Harris wrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; I'm wanting to grab the information out in chunks, so createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; would be what my regex would grab. I'm currently only able to grab out the first line and part of the second line, but no more. regex is as follows my_regexp = re.compile(createNode\ animCurve.*\n[\t*setAttr.*\n]*) I've run several variations of this, but none return me all of the expected information. Is there something special that needs to be done to have the regexp grab any number of the setAttr lines without specification? Groups are marked with parens (...) not brackets [...]. text = \ ... createNode animCurveTU -n test:master_globalSmooth; ... setAttr .tan 9; ... setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; ... setAttr -s 4 .kit[3] 10; ... setAttr -s 4 .kot[3] 10; ... createNode animCurveTU -n test:master_res; ... setAttr .tan 9; ... setAttr .ktv[0] 103 0; ... setAttr .kot[0] 5; ... createNode animCurveTU -n test:master_faceRig; ... setAttr .tan 9; ... setAttr .ktv[0] 103 0; ... setAttr .kot[0] 5; ... for m in re.compile((createNode animCurve.*\n(\s*setAttr.*\n)*)).finditer(text): ... print m.group(1) ... print - * 40 ... createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: RE: Multiline regex
I could make it that simple, but that is also incredibly slow and on a file with several million lines, it takes somewhere in the league of half an hour to grab all the data. I need this to grab data from many many file and return the data quickly. Brandon L. Harris Andreas Tawn wrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: createNode animCurveTU -n test:master_globalSmooth; setAttr .tan 9; setAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0; setAttr -s 4 .kit[3] 10; setAttr -s 4 .kot[3] 10; createNode animCurveTU -n test:master_res; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; I'm wanting to grab the information out in chunks, so createNode animCurveTU -n test:master_faceRig; setAttr .tan 9; setAttr .ktv[0] 103 0; setAttr .kot[0] 5; would be what my regex would grab. I'm currently only able to grab out the first line and part of the second line, but no more. regex is as follows my_regexp =e.compile(createNode\ animCurve.*\n[\t*setAttr.*\n]*) I've run several variations of this, but none return me all of the expected information. Is there something special that needs to be done to have the regexp grab any number of the setAttr lines without specification? Brandon L. Harris Aren't you making life too complicated for yourself? blocks =] for line in yourFile: if line.startswith(createNode): if currentBlock: blocks.append(currentBlock) currentBlock =line] else: currentBlock.append(line) blocks.append(currentBlock) Cheers, Drea -- http://mail.python.org/mailman/listinfo/python-list
RE: RE: Multiline regex
I could make it that simple, but that is also incredibly slow and on a file with several million lines, it takes somewhere in the league of half an hour to grab all the data. I need this to grab data from many many file and return the data quickly. Brandon L. Harris That's surprising. I just made a file with 13 million lines of your data (447Mb) and read it with my code. It took a little over 36 seconds. There must be something different in your set up or the real data you've got. Cheers, Drea -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
Could it be that there isn't just that type of data in the file? there are many different types, that is just one that I'm trying to grab. Brandon L. Harris Andreas Tawn wrote: I could make it that simple, but that is also incredibly slow and on a file with several million lines, it takes somewhere in the league of half an hour to grab all the data. I need this to grab data from many many file and return the data quickly. Brandon L. Harris That's surprising. I just made a file with 13 million lines of your data (447Mb) and read it with my code. It took a little over 36 seconds. There must be something different in your set up or the real data you've got. Cheers, Drea -- http://mail.python.org/mailman/listinfo/python-list
RE: Multiline regex
I could make it that simple, but that is also incredibly slow and on a file with several million lines, it takes somewhere in the league of half an hour to grab all the data. I need this to grab data from many many file and return the data quickly. Brandon L. Harris That's surprising. I just made a file with 13 million lines of your data (447Mb) and read it with my code. It took a little over 36 seconds. There must be something different in your set up or the real data you've got. Cheers, Drea Could it be that there isn't just that type of data in the file? there are many different types, that is just one that I'm trying to grab. Brandon L. Harris I don't see why it would make such a difference. If your data looks like... block header \tattribute \tattribute \tattribute Just change this line... if line.startswith(createNode): to... if not line.startswith(\t): and it won't care what sort of data the file contains. Processing that data after you've collected it will still take a while, but that's the same whichever method you use to read it. Cheers, Drea p.s. Just noticed I hadn't pre-declared the currentBlock list. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
Brandon Harris wrote: I'm trying to read in and parse an ascii type file that contains information that can span several lines. Example: What about something like this (you need re.MULTILINE): In [16]: re.findall('^([^ ].*\n([ ].*\n)+)', a, re.MULTILINE) Out[16]: [('createNode animCurveTU -n test:master_globalSmooth;\nsetAttr .tan 9;\nsetAttr -s 4 .ktv[0:3] 101 0 163 0 169 0 201 0;\nsetAttr -s 4 .kit[3] 10;\nsetAttr -s 4 .kot[3] 10;\n', 'setAttr -s 4 .kot[3] 10;\n'), ('createNode animCurveTU -n test:master_res;\nsetAttr .tan 9;\n setAttr .ktv[0] 103 0;\nsetAttr .kot[0] 5;\n', 'setAttr .kot[0] 5;\n'), ('createNode animCurveTU -n test:master_faceRig;\nsetAttr .tan 9;\n setAttr .ktv[0] 103 0;\n', 'setAttr .ktv[0] 103 0;\n')] If you blocks start without a space and subsequent lines with a space. Jeremy -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex
On Wed, 21 Jul 2010 10:06:14 -0500, Brandon Harris wrote: what do you mean by slurp the entire file? I'm trying to use regular expressions because line by line parsing will be too slow. And example file would have somewhere in the realm of 6 million lines of code. And you think trying to run a regex over all 6 million lines at once will be faster? I think you're going to be horribly, horribly disappointed. And then on Wed, 21 Jul 2010 10:42:11 -0500, Brandon Harris wrote: I could make it that simple, but that is also incredibly slow and on a file with several million lines, it takes somewhere in the league of half an hour to grab all the data. I need this to grab data from many many file and return the data quickly. What do you mean grab all the data? If all you mean is read the file, then 30 minutes to read ~ 100MB of data is incredibly slow and you're probably doing something wrong, or you're reading it over a broken link with very high packet loss, or something. If you mean read the data AND parse it, then whether that is incredibly slow or amazingly fast depends entirely on how complicated your parser needs to be. If *all* you mean is read the file and group the lines, for later processing, then I would expect it to take well under a minute to group millions of lines. Here's a simulation I ran, using 2001000 lines of text based on the examples you gave. It grabs the blocks, as required, but does no further parsing of them. def merge(lines): Join multiple lines into a single block. accumulator = [] for line in lines: if line.lower().startswith('createnode'): if accumulator: yield ''.join(accumulator) accumulator = [] accumulator.append(line) if accumulator: yield ''.join(accumulator) def test(): import time t = time.time() count = 0 f = open('/steve/test.junk') for block in merge(f): # do some make-work n = sum([1 for c in block if c in '1234567890']) count += 1 print Processed %d blocks from 2M+ lines. % count print Time taken:, time.time() - t, seconds And the result on a low-end PC: test() Processed 1000 blocks from 2M+ lines. Time taken: 17.4497909546 seconds -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Multiline regex help
Hey Folks, I've got some info in a bunch of files that kind of looks like so: Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 and so on... Anyhow, these fields repeat several times in a given file (number of repetitions varies from file to file). The number on the line following the RelevantInfo lines is really what I'm after. Ideally, I would like to have something like so: RelevantInfo1 = 10/10/04 # The variable name isn't actually important RelevantInfo3 = 23 # it's just there to illustrate what info I'm # trying to snag. Score[RelevantInfo1][RelevantInfo3] = 22 # The value from RelevantInfo2 Collected from all of the files. So, there would be several of these scores per file and there are a bunch of files. Ultimately, I am interested in printing them out as a csv file but that should be relatively easy once they are trapped in my array of doom cue evil laughter. I've got a fairly ugly solution (I am using this term *very* loosely) using awk and his faithfail companion sed, but I would prefer something in python. Thanks for your time. -- McGowan's Madison Avenue Axiom: If an item is advertised as under $50, you can bet it's not $19.95. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Yatima wrote: Hey Folks, I've got some info in a bunch of files that kind of looks like so: Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 and so on... Anyhow, these fields repeat several times in a given file (number of repetitions varies from file to file). The number on the line following the RelevantInfo lines is really what I'm after. Ideally, I would like to have something like so: RelevantInfo1 = 10/10/04 # The variable name isn't actually important RelevantInfo3 = 23 # it's just there to illustrate what info I'm # trying to snag. Here is a way to create a list of [RelevantInfo, value] pairs: import cStringIO raw_data = '''Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34''' raw_data = cStringIO.StringIO(raw_data) data = [] for line in raw_data: if line.startswith('RelevantInfo'): key = line.strip() value = raw_data.next().strip() data.append([key, value]) print data Score[RelevantInfo1][RelevantInfo3] = 22 # The value from RelevantInfo2 I'm not sure what you mean by this. Do you want to build a Score dictionary as well? Kent Collected from all of the files. So, there would be several of these scores per file and there are a bunch of files. Ultimately, I am interested in printing them out as a csv file but that should be relatively easy once they are trapped in my array of doom cue evil laughter. I've got a fairly ugly solution (I am using this term *very* loosely) using awk and his faithfail companion sed, but I would prefer something in python. Thanks for your time. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Yatima wrote: Hey Folks, I've got some info in a bunch of files that kind of looks like so: Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 and so on... Anyhow, these fields repeat several times in a given file (number of repetitions varies from file to file). The number on the line following the RelevantInfo lines is really what I'm after. Ideally, I would like to have something like so: RelevantInfo1 = 10/10/04 # The variable name isn't actually important RelevantInfo3 = 23 # it's just there to illustrate what info I'm # trying to snag. Score[RelevantInfo1][RelevantInfo3] = 22 # The value from RelevantInfo2 A possible solution, using the re module: py s = \ ... Gibberish ... 53 ... MoreGarbage ... 12 ... RelevantInfo1 ... 10/10/04 ... NothingImportant ... ThisDoesNotMatter ... 44 ... RelevantInfo2 ... 22 ... BlahBlah ... 343 ... RelevantInfo3 ... 23 ... Hubris ... Crap ... 34 ... py import re py m = re.compile(r^RelevantInfo1\n([^\n]*) ....* ...^RelevantInfo2\n([^\n]*) ....* ...^RelevantInfo3\n([^\n]*), ...re.DOTALL | re.MULTILINE | re.VERBOSE) py score = {} py for info1, info2, info3 in m.findall(s): ... score.setdefault(info1, {})[info3] = info2 ... py score {'10/10/04': {'23': '22'}} Note that I use DOTALL to allow .* to cross line boundaries, MULTILINE to have ^ apply at the start of each line, and VERBOSE to allow me to write the re in a more readable form. If I didn't get your dict update quite right, hopefully you can see how to fix it! HTH, STeVe -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
On Thu, 03 Mar 2005 09:54:02 -0700, Steven Bethard [EMAIL PROTECTED] wrote: A possible solution, using the re module: py s = \ ... Gibberish ... 53 ... MoreGarbage ... 12 ... RelevantInfo1 ... 10/10/04 ... NothingImportant ... ThisDoesNotMatter ... 44 ... RelevantInfo2 ... 22 ... BlahBlah ... 343 ... RelevantInfo3 ... 23 ... Hubris ... Crap ... 34 ... py import re py m = re.compile(r^RelevantInfo1\n([^\n]*) ....* ...^RelevantInfo2\n([^\n]*) ....* ...^RelevantInfo3\n([^\n]*), ...re.DOTALL | re.MULTILINE | re.VERBOSE) py score = {} py for info1, info2, info3 in m.findall(s): ... score.setdefault(info1, {})[info3] = info2 ... py score {'10/10/04': {'23': '22'}} Note that I use DOTALL to allow .* to cross line boundaries, MULTILINE to have ^ apply at the start of each line, and VERBOSE to allow me to write the re in a more readable form. If I didn't get your dict update quite right, hopefully you can see how to fix it! Thanks! That was very helpful. Unfortunately, I wasn't completely clear when describing the problem. Is there anyway to extract multiple scores from the same file and from multiple files (I will probably use the fileinput module to deal with multiple files). So, if I've got say: Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 SecondSetofGarbage 2423 YouGetThePicture 342342 RelevantInfo1 10/10/04 HoHum 343 MoreStuffNotNeeded 232 RelevantInfo2 33 RelevantInfo3 44 sdfsdf RelevantInfo1 10/11/04 InsertBoringFillerHere 43234 Stuff MoreStuff RelevantInfo2 45 ExcitingIsntIt 324234 RelevantInfo3 60 Lalala Sorry for the long and painful example input. Notice that the first two RelevantInfo1 fields have the same info but that the RelevantInfo2 and RelevantInfo3 fields have different info. Also, there will be cases where RelevantInfo3 might be the same with a different RelevantInfo2. What, I'm hoping for is something along then lines of being able to organize it like so (don't worry about the format of the output -- I'll deal with that later; RelevantInfo shortened to Info for readability): Info1[0], Info[1],Info[2] ... Info3[0]Info2[Info1[0],Info3[0]]Info2[Info1[1],Info3[1]]... Info3[1]Info2[Info1[0],Info3[1]]... Info3[2]Info2[Info1[0],Info3[2]]... ... I don't really care if it's a list, dictionary, array etc. Thanks again for your help. The multiline option in the re module is very useful. Take care. -- Clarke's Conclusion: Never let your sense of morals interfere with doing the right thing. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Have a look at martel, part of biopython. The world of bioinformatics is filled with files with structure like this. http://www.biopython.org/docs/api/public/Martel-module.html James On Thursday 03 March 2005 12:03 pm, Yatima wrote: On Thu, 03 Mar 2005 09:54:02 -0700, Steven Bethard [EMAIL PROTECTED] wrote: A possible solution, using the re module: py s = \ ... Gibberish ... 53 ... MoreGarbage ... 12 ... RelevantInfo1 ... 10/10/04 ... NothingImportant ... ThisDoesNotMatter ... 44 ... RelevantInfo2 ... 22 ... BlahBlah ... 343 ... RelevantInfo3 ... 23 ... Hubris ... Crap ... 34 ... py import re py m = re.compile(r^RelevantInfo1\n([^\n]*) ....* ...^RelevantInfo2\n([^\n]*) ....* ...^RelevantInfo3\n([^\n]*), ...re.DOTALL | re.MULTILINE | re.VERBOSE) py score = {} py for info1, info2, info3 in m.findall(s): ... score.setdefault(info1, {})[info3] = info2 ... py score {'10/10/04': {'23': '22'}} Note that I use DOTALL to allow .* to cross line boundaries, MULTILINE to have ^ apply at the start of each line, and VERBOSE to allow me to write the re in a more readable form. If I didn't get your dict update quite right, hopefully you can see how to fix it! Thanks! That was very helpful. Unfortunately, I wasn't completely clear when describing the problem. Is there anyway to extract multiple scores from the same file and from multiple files (I will probably use the fileinput module to deal with multiple files). So, if I've got say: Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 SecondSetofGarbage 2423 YouGetThePicture 342342 RelevantInfo1 10/10/04 HoHum 343 MoreStuffNotNeeded 232 RelevantInfo2 33 RelevantInfo3 44 sdfsdf RelevantInfo1 10/11/04 InsertBoringFillerHere 43234 Stuff MoreStuff RelevantInfo2 45 ExcitingIsntIt 324234 RelevantInfo3 60 Lalala Sorry for the long and painful example input. Notice that the first two RelevantInfo1 fields have the same info but that the RelevantInfo2 and RelevantInfo3 fields have different info. Also, there will be cases where RelevantInfo3 might be the same with a different RelevantInfo2. What, I'm hoping for is something along then lines of being able to organize it like so (don't worry about the format of the output -- I'll deal with that later; RelevantInfo shortened to Info for readability): Info1[0], Info[1],Info[2] ... Info3[0]Info2[Info1[0],Info3[0]]Info2[Info1[1],Info3[1]]... Info3[1]Info2[Info1[0],Info3[1]]... Info3[2]Info2[Info1[0],Info3[2]]... ... I don't really care if it's a list, dictionary, array etc. Thanks again for your help. The multiline option in the re module is very useful. Take care. -- Clarke's Conclusion: Never let your sense of morals interfere with doing the right thing. -- James Stroud, Ph.D. UCLA-DOE Institute for Genomics and Proteomics Box 951570 Los Angeles, CA 90095 -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
On Thu, 03 Mar 2005 07:14:50 -0500, Kent Johnson [EMAIL PROTECTED] wrote: Here is a way to create a list of [RelevantInfo, value] pairs: import cStringIO raw_data = '''Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34''' raw_data = cStringIO.StringIO(raw_data) data = [] for line in raw_data: if line.startswith('RelevantInfo'): key = line.strip() value = raw_data.next().strip() data.append([key, value]) print data Thank you. This isn't exactly what I'm looking for (I wasn't clear in describing the problem -- please see my reply to Steve for a, hopefully, better explanation) but it does give me a few ideas. Score[RelevantInfo1][RelevantInfo3] = 22 # The value from RelevantInfo2 I'm not sure what you mean by this. Do you want to build a Score dictionary as well? Sure... Uhhh.. I think. Okay, what I want is some kind of awk-like associative array because the raw data files will have repeats for certain field vaues such that there would be, for example, multiple RelevantInfo2's and RelevantInfo3's for the same RelevantInfo1 (i.e. on the same date). To make matters more exciting, there will be multiple RelevantInfo1's (dates) for the same RelevantInfo3 (e.g. a subject ID). RelevantInfo2 will be the value for all unique combinations of RelevantInfo1 and RelevantInfo3. There will be multiple occurrences of these fields in the same file (original data sample was not very good for this reason) and multiple files as well. The interesting three fields will always be repeated in the same order although the amount of irrelevant data in between may vary. So: RelevantInfo1 10/10/04 snipped crap RelevantInfo2 12 more snippage RelevantInfo3 43 more snippage RelevantInfo1 10/10/04- The same as the first occurrence of RelevantInfo1 snipped RelevantInfo2 22 snipped RelevantInfo3 25 snipped RelevantInfo1 10/11/04 snipped RelevantInfo2 34 snipped RelevantInfo3 28 snipped RelevantInfo1 10/12/04 snipped RelevantInfo2 98 snipped RelevantInfo3 25- The same as the second occurrence of RelevantInfo3 ... Sorry for the long and tedious data example. There will be missing values for some combinations of RelevantInfo1 and RelevantInfo3 so hopefully that won't be an issue. Thanks again for your reply. Take care. -- I figured there was this holocaust, right, and the only ones left alive were Donna Reed, Ozzie and Harriet, and the Cleavers. -- Wil Wheaton explains why everyone in Star Trek: The Next Generation is so nice -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
I found the original paper for Martel: http://www.dalkescientific.com/Martel/ipc9/ On Thursday 03 March 2005 12:26 pm, James Stroud wrote: Have a look at martel, part of biopython. The world of bioinformatics is filled with files with structure like this. http://www.biopython.org/docs/api/public/Martel-module.html James On Thursday 03 March 2005 12:03 pm, Yatima wrote: -- James Stroud, Ph.D. UCLA-DOE Institute for Genomics and Proteomics Box 951570 Los Angeles, CA 90095 -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Yatima wrote: On Thu, 03 Mar 2005 09:54:02 -0700, Steven Bethard [EMAIL PROTECTED] wrote: A possible solution, using the re module: py s = \ ... Gibberish ... 53 ... MoreGarbage ... 12 ... RelevantInfo1 ... 10/10/04 ... NothingImportant ... ThisDoesNotMatter ... 44 ... RelevantInfo2 ... 22 ... BlahBlah ... 343 ... RelevantInfo3 ... 23 ... Hubris ... Crap ... 34 ... py import re py m = re.compile(r^RelevantInfo1\n([^\n]*) ....* ...^RelevantInfo2\n([^\n]*) ....* ...^RelevantInfo3\n([^\n]*), ...re.DOTALL | re.MULTILINE | re.VERBOSE) py score = {} py for info1, info2, info3 in m.findall(s): ... score.setdefault(info1, {})[info3] = info2 ... py score {'10/10/04': {'23': '22'}} Note that I use DOTALL to allow .* to cross line boundaries, MULTILINE to have ^ apply at the start of each line, and VERBOSE to allow me to write the re in a more readable form. If I didn't get your dict update quite right, hopefully you can see how to fix it! Thanks! That was very helpful. Unfortunately, I wasn't completely clear when describing the problem. Is there anyway to extract multiple scores from the same file and from multiple files I think if you use the non-greedy .*? instead of the greedy .*, you'll get this behavior. For example: py s = \ ... Gibberish ... 53 ... MoreGarbage [snip a whole bunch of stuff] ... RelevantInfo3 ... 60 ... Lalala ... py import re py m = re.compile(r^RelevantInfo1\n([^\n]*) ....*? ...^RelevantInfo2\n([^\n]*) ....*? ...^RelevantInfo3\n([^\n]*), ...re.DOTALL | re.MULTILINE | re.VERBOSE) py score = {} py for info1, info2, info3 in m.findall(s): ... score.setdefault(info1, {})[info3] = info2 ... py score {'10/10/04': {'44': '33', '23': '22'}, '10/11/04': {'60': '45'}} If you might have multiple info2 values for the same (info1, info3) pair, you can try something like: py score = {} py for info1, info2, info3 in m.findall(s): ... score.setdefault(info1, {}).setdefault(info3, []).append(info2) ... py score {'10/10/04': {'44': ['33'], '23': ['22']}, '10/11/04': {'60': ['45']}} HTH, STeVe -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Here is another attempt. I'm still not sure I understand what form you want the data in. I made a dict - dict - list structure so if you lookup e.g. scores['10/11/04']['60'] you get a list of all the RelevantInfo2 values for Relevant1='10/11/04' and Relevant2='60'. The parser is a simple-minded state machine that will misbehave if the input does not have entries in the order Relevant1, Relevant2, Relevant3 (with as many intervening lines as you like). All three values are available when Relevant3 is detected so you could do something else with them if you want. HTH Kent import cStringIO raw_data = '''Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 Gibberish 53 MoreGarbage 12 RelevantInfo1 10/10/04 NothingImportant ThisDoesNotMatter 44 RelevantInfo2 22 BlahBlah 343 RelevantInfo3 23 Hubris Crap 34 SecondSetofGarbage 2423 YouGetThePicture 342342 RelevantInfo1 10/10/04 HoHum 343 MoreStuffNotNeeded 232 RelevantInfo2 33 RelevantInfo3 44 sdfsdf RelevantInfo1 10/11/04 InsertBoringFillerHere 43234 Stuff MoreStuff RelevantInfo2 45 ExcitingIsntIt 324234 RelevantInfo3 60 Lalala''' raw_data = cStringIO.StringIO(raw_data) scores = {} info1 = info2 = info3 = None for line in raw_data: if line.startswith('RelevantInfo1'): info1 = raw_data.next().strip() elif line.startswith('RelevantInfo2'): info2 = raw_data.next().strip() elif line.startswith('RelevantInfo3'): info3 = raw_data.next().strip() scores.setdefault(info1, {}).setdefault(info3, []).append(info2) info1 = info2 = info3 = None print scores print scores['10/11/04']['60'] print scores['10/10/04']['23'] ## prints: {'10/10/04': {'44': ['33'], '23': ['22', '22']}, '10/11/04': {'60': ['45']}} ['45'] ['22', '22'] -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
On Thu, 03 Mar 2005 16:25:39 -0500, Kent Johnson [EMAIL PROTECTED] wrote: Here is another attempt. I'm still not sure I understand what form you want the data in. I made a dict - dict - list structure so if you lookup e.g. scores['10/11/04']['60'] you get a list of all the RelevantInfo2 values for Relevant1='10/11/04' and Relevant2='60'. The parser is a simple-minded state machine that will misbehave if the input does not have entries in the order Relevant1, Relevant2, Relevant3 (with as many intervening lines as you like). All three values are available when Relevant3 is detected so you could do something else with them if you want. HTH Kent import cStringIO raw_data = '''Gibberish 53 MoreGarbage [mass snippage] 60 Lalala''' raw_data = cStringIO.StringIO(raw_data) scores = {} info1 = info2 = info3 = None for line in raw_data: if line.startswith('RelevantInfo1'): info1 = raw_data.next().strip() elif line.startswith('RelevantInfo2'): info2 = raw_data.next().strip() elif line.startswith('RelevantInfo3'): info3 = raw_data.next().strip() scores.setdefault(info1, {}).setdefault(info3, []).append(info2) info1 = info2 = info3 = None print scores print scores['10/11/04']['60'] print scores['10/10/04']['23'] ## prints: {'10/10/04': {'44': ['33'], '23': ['22', '22']}, '10/11/04': {'60': ['45']}} ['45'] ['22', '22'] Thank you so much. Your solution and Steve's both give me what I'm looking for. I appreciate both of your incredibly quick replies! Take care. -- You worry too much about your job. Stop it. You are not paid enough to worry. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
On Thu, 3 Mar 2005 12:26:37 -0800, James Stroud [EMAIL PROTECTED] wrote: Have a look at martel, part of biopython. The world of bioinformatics is filled with files with structure like this. http://www.biopython.org/docs/api/public/Martel-module.html James Thanks for the link. Steve and Kent have provided me with nice solutions but I will check this out anyways for future referenced. Take care. -- You may easily play a joke on a man who likes to argue -- agree with him. -- Ed Howe -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Kent Johnson wrote: for line in raw_data: if line.startswith('RelevantInfo1'): info1 = raw_data.next().strip() elif line.startswith('RelevantInfo2'): info2 = raw_data.next().strip() elif line.startswith('RelevantInfo3'): info3 = raw_data.next().strip() scores.setdefault(info1, {}).setdefault(info3, []).append(info2) info1 = info2 = info3 = None Very pretty. =) I have to say, I hadn't ever used iterators this way before, that is, calling their next method from within a for-loop. I like it. =) Thanks for opening my mind. ;) STeVe -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiline regex help
Steven Bethard wrote: Kent Johnson wrote: for line in raw_data: if line.startswith('RelevantInfo1'): info1 = raw_data.next().strip() elif line.startswith('RelevantInfo2'): info2 = raw_data.next().strip() elif line.startswith('RelevantInfo3'): info3 = raw_data.next().strip() scores.setdefault(info1, {}).setdefault(info3, []).append(info2) info1 = info2 = info3 = None Very pretty. =) I have to say, I hadn't ever used iterators this way before, that is, calling their next method from within a for-loop. I like it. =) I confess I have a nagging suspicion that someone who actually knows something about CPython internals will tell me why it's a bad idea...but it sure is handy! Thanks for opening my mind. ;) My pleasure :-) Kent -- http://mail.python.org/mailman/listinfo/python-list