Hi all, Although I have a number of decent text-editors on both my Mac and PC, I often find myself cleaning up my market data files in VI or VIM in a unix shell on my Mac. I'm not sure if anyone else is or should be, but for the benefit of newcomers I'd thought I'd post a couple of helpful things.
Firstly, I have a multi-platform environment, so sometimes, it is necessary to change line endings from DOS to UNIX: I use this global search/replace in VI to do this: [when in vi] :g/\r/s///g Also, sometimes, I get duplicate timestamps in my datafiles. This may be unique to my setup and it is getting rarer, but when it happens it is really annoying because the JBT backtester will stop and complain about each duplicate timestamp individually and I end up going back and forth between editing the datafile and running a backtest. So, I found the following vi expression will let me quickly jump to the offending line so I don't require JBT to tell me which line it is. [when in vi] :g/^......,\(......\).*\n......,\1 Sure, I could probably make it a bit shorter with some complicated looking grep expressions, but this is very quick (and mindless) to type. If you are purely on a Windows machine, you may prefer to keep the DOS line-endings, in which case the above would be: [when in vi] :g/^......,\(......\).*\r\n......,\1 Anyway, then, after I've fixed it, I can just press "/" and <return> to repeat the last search and get quickly to the next one, if any. Now, I generally like to look at the line to determine the best course of action. But if you wanted to simply nuke ALL the lines with the duplicate timestamps in one go, you could use: [when in vi] :g/^......,\(......\).*\n......,\1/s/^.*\n//g Also, I often use SED to trim all the lines that are before 08:00:00 and after 16:59:59 each day since I don't need them and they will only make backtesting and optimization take a bit longer. [at the shell prompt] sed -e '/^......,0[0-7]/d' -e '/^......,1[789]/d' -e '/^......,2/d' INFILE.TXT > OUTFILE.TXT Anyway, I hope this helps someone. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en -~----------~----~----~----~------~----~------~--~---
