Thanks for sharing this! On Friday, August 21, 2009, MKoistinen <[email protected]> wrote: > > Hi all, > > Although I have a number of decent text-editors on both my Mac and PC, > I often find myself cleaning up my market data files in VI or VIM in a > unix shell on my Mac. I'm not sure if anyone else is or should be, > but for the benefit of newcomers I'd thought I'd post a couple of > helpful things. > > Firstly, I have a multi-platform environment, so sometimes, it is > necessary to change line endings from DOS to UNIX: I use this global > search/replace in VI to do this: > > [when in vi] > :g/\r/s///g > > Also, sometimes, I get duplicate timestamps in my datafiles. This may > be unique to my setup and it is getting rarer, but when it happens it > is really annoying because the JBT backtester will stop and complain > about each duplicate timestamp individually and I end up going back > and forth between editing the datafile and running a backtest. > > So, I found the following vi expression will let me quickly jump to > the offending line so I don't require JBT to tell me which line it is. > > [when in vi] > :g/^......,\(......\).*\n......,\1 > > Sure, I could probably make it a bit shorter with some complicated > looking grep expressions, but this is very quick (and mindless) to > type. If you are purely on a Windows machine, you may prefer to keep > the DOS line-endings, in which case the above would be: > > [when in vi] > :g/^......,\(......\).*\r\n......,\1 > > Anyway, then, after I've fixed it, I can just press "/" and <return> > to repeat the last search and get quickly to the next one, if any. > > Now, I generally like to look at the line to determine the best course > of action. But if you wanted to simply nuke ALL the lines with the > duplicate timestamps in one go, you could use: > > [when in vi] > :g/^......,\(......\).*\n......,\1/s/^.*\n//g > > Also, I often use SED to trim all the lines that are before 08:00:00 > and after 16:59:59 each day since I don't need them and they will only > make backtesting and optimization take a bit longer. > > [at the shell prompt] > sed -e '/^......,0[0-7]/d' -e '/^......,1[789]/d' -e '/^......,2/d' > INFILE.TXT > OUTFILE.TXT > > Anyway, I hope this helps someone. > > >
--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "JBookTrader" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/jbooktrader?hl=en -~----------~----~----~----~------~----~------~--~---
