> I might be making to big a deal over this but I'm new to trading >but a > bit of a fusspot over having correct data. Also the system I am >trying > to write is highly dependent on volume,
I put a lot of effort into learning data too.... I tailed off it after a while. It's ok to go into detail, in fact I think you have the attributes that are required for this game: attention to detail, self-motivated, persistent, going outside the box etc. Also, IMO, it is OK to forget about the details AFTER you know what the detail involves and can make a conscious decision to live with some approximations, and not BEFORE (which some do). I am heading towards a summer break so I might not be around. --- In [email protected], "elegalpublishing" <mark.a.br...@...> wrote: > > Hi Brian: > > Sorry for taking so long to respond, thanks for checking this out. > > I will also run my routines over the 25/26 Feb also to see if I get > similar results and will email you what i get. > > I might be making to big a deal over this but I'm new to trading but a > bit of a fusspot over having correct data. Also the system I am trying > to write is highly dependent on volume, > > Cheers > Mark > > --- In [email protected], "brian_z111" <brian_z111@> wrote: > > > > Mark, > > > > I checked my files. > > I have 6 mths tick data ending Feb08. > > > > I looked at two files - 25 and 26/Feb. > > > > They exceed Xcel max rows. > > I remember now that I had to work on them in txt format. > > > > I also have the ASX file format doc ... do you have that .. I can > > email it? > > > > My best guess is: > > > > - you would get better advice from Weblink or ASX :-) > > - but! as long as you delete all errors, and cancellation pairs, then > > you can import all the quotes and AB will filter the view for you, as > > per your settings. > > - you will see some wierd and wonderful prices, in the pre, but that > > is normal for 24 hr stock charts and you soon learn to ignore low vol > > wierd prices > > - in 24 hr view I put a horizontal line on the chart to mark the > > formal open and close of market hours > > > > > > > > If you want to go into detail: > > > > - the premarket totals around 3% vol in my file > > - it contains housekeeping stuff e.g. warrants, executed puts and > > calls etc > > - the prices in the pre have no relevance to the current day or the > > prev day e.g. puts executing for $25 -29 when BHP traded between > > 22.40-22.90 .... BHP hadn't traded at >$25 in the 20 days prior. > > > > - a lot of the pre quotes are dated for the previous day > > > > - I think the pre market quotes are not important ... you can keep > > them, just in case, if you want to ... you will probably never look > > at them .. dumping them isn't high risk ... I'm 99% sure they are of > > no use (based on my two files anyway). > > > > - errors are limited to (???) quotes with no price or vol ... delete > > all of them ... most of the errors at in the pre or afters > > > > - cancellations are not present in large numbers ... they are C > > trades matched to the trade above by trade number ... delete the > > pairs .. most are in market hours .. a few in the pre > > > > > > > > From the format doc: > > > > A trade cancellation consists of the following CSV record: > > > > C, Symbol, Date, TS Number, Cancelled TS Number, Cancelled Date, > > Cancel Reason > > Where: Symbol -- Unique security identifier, no more than 6 > > characters. The first three characters indicate the issuer or > > underlying security. > > Date -- The date that the trade cancellation occurred. > > TS Number -- Trade sequence number. This is a 6 digit identifier for > > the trade cancellation, which is unique for the current day. > > Cancelled TS Number -- The TS number of the trade being cancelled. > > This, coupled with the cancelled date field, can be used to uniquely > > identify the trade. > > Cancelled Date -- The date on which the trade being cancelled > > occurred. > > Cancel Reason -- The reason for the cancellation, being one of the > > following: > > Incorrect Broker Number > > Incorrect Volume > > Incorrect Price > > Incorrect Security Code > > Other > > Data Entry > > Stop Cancellation Request > > > > Example: > > > > XT is a cross trade > > Yahoo has wrapped the 2nd line > > the C line cancels trade 12334 above > > I checked approx 50 and they are all matching pairs on two lines > > > > T BHP 20080226 100544 12334 39.26 30000 > > 20080229 XT > > C BHP 20080226 12334 12334 20080226 > > > > > > - none of the C's in my files gave a reason ... the intitial trades > > were nearly all crossed trades though > > > > - trading continues beyond 1600hrs for 15 - 30 mins ... you can keep > > this in the database if you want ... it is good data ... in fact best > > that you do keep it > > > > - the after market quotes include a lot of errors > > > > > > > > > > > > > > > > --- In [email protected], "brian_z111" <brian_z111@> wrote: > > > > > > Yes, that is the area I am referring to. > > > > > > Importing 24 hours data would be the best, but then you need to be > > > 100% certain about the stuff that is in the after hours timeslot, > > > because that is where the data is non-typical. > > > > > > > > > I'll get my file out and check ... my file is approx one year > > old.... > > > might be different to yours. > > > > > > It's worth it since it has relevance to other Aussie traders. > > > > > > Better still if we could leave some brief notes, on a method, in > > the > > > file section but that isn't a pre-requisite if time presses. > > > > > > I'll come back to you ... if not today then within a few days. > > > > > > > > > --- In [email protected], "elegalpublishing" > > > <mark.a.brand@> wrote: > > > > > > > > Hi Brian: > > > > > > > > <snip> > > > > I'm not certain but I think a trade for 10000 as a buy might be > > > > corrected as a sell the next day, so both orders should be > > cancelled > > > > for a 5 star database. > > > > </snip> > > > > > > > > Thanks for that, there are two types of transactions in the > > Weblink > > > > tick data, "T" normal trade and "C" for cancelled trades. Would > > you > > > > expect the situation you refer to above be covered by the > > > cancellation > > > > transactions. > > > > > > > > Thanks > > > > Mark > > > > > > > > PS. I will send the scripts to your email. > > > > > > > > --- In [email protected], "brian_z111" <brian_z111@> > > wrote: > > > > > > > > > > Have a quick look to make sure you are not double counting vol > > or > > > > > data ... I don't have a file in front of me but from memory > > they > > > > > trades have a ref number or order number .... I'm not certain > > but > > > I > > > > > think a trade for 10000 as a buy might be corrected as a sell > > the > > > > > next day, so both orders should be cancelled for a 5 star > > > database. > > > > > > > > > > > > > > > --- In [email protected], "brian_z111" <brian_z111@> > > > wrote: > > > > > > > > > > > > I guess we are better off to trust the ASX file .... so if we > > > > > assume > > > > > > their corrections are spot on, and you enter all clean 24 > > hour > > > data > > > > > > by date (sounds like you have done that) then just use AB's > > > built > > > > > in > > > > > > database settings >> intraday settings to view 24 hours or > > > market > > > > > > hours only ... this works fine for me and you can have the > > best > > > of > > > > > > both worlds. > > > > > > > > > > > > > > > > > > > gawk and bash under cygwin, i can send the scripts to you > > if > > > you > > > > > are > > > > > > > interested. > > > > > > > > > > > > Thanks that would be good - might help me with my programming > > > > > > education. > > > > > > > > > > > > brian_z111|asat|yahoo.com > > > > > > > > > > > > Interesting example in the US, last night, of how the goal > > > posts > > > > > are > > > > > > shifting .... formerly a daily bar was market hour data only > > > and > > > > > > nothing else mattered. > > > > > > > > > > > > Citigroup (C) traded up from 3.48 to 4.52 in overnight > > > trading ... > > > > > > then drifted a bit to open at 4.29 and then traded sideways > > > during > > > > > > market hours to finish at 4.17 > > > > > > > > > > > > This only happens in the hot US stocks but it changes the > > > meaning > > > > > of > > > > > > daily bars. > > > > > > > > > > > > C is in the top ten US stocks for vol at the moment and this > > is > > > > > where > > > > > > the O'nite action seems to happen. > > > > > > > > > > > > (one the US opens this list will be populated with high vol > > > stocks) > > > > > > > > > > > > http://finance.yahoo.com/actives?e=us > > > > > > > > > > > > The open gap on the intraday chart was actually o'nite trade. > > > > > > > > > > > > http://finance.yahoo.com/q/bc?s=C&t=5d > > > > > > > > > > > > > > > > > > > > > > > > --- In [email protected], "elegalpublishing" > > > > > > <mark.a.brand@> wrote: > > > > > > > > > > > > > > Hi Brian: > > > > > > > > > > > > > > Thanks a lot for your response, its good because you made > > me > > > > > > actually > > > > > > > count the out of hours volume which i had not actually done > > > > > prior. > > > > > > As > > > > > > > you found it is about 2% which I can live with. > > > > > > > > > > > > > > <snip> > > > > > > > I seem to recall adding the out of hours vol and it came > > to > > > > > > approx > > > > > > > <2% of daily vol ... is that what you get? > > > > > > > </snip> > > > > > > > > > > > > > > One example i did was 4M on 114M total volume > > > > > > > > > > > > > > <snip> > > > > > > > are you massaging the data into 1 min bars before > > importing > > > > > into > > > > > > AB? > > > > > > > </snip> > > > > > > > > > > > > > > yes > > > > > > > > > > > > > > <snip> > > > > > > > - briefly what language are you using and how are you going > > > about > > > > > > it? > > > > > > > </snip> > > > > > > > > > > > > > > gawk and bash under cygwin, i can send the scripts to you > > if > > > you > > > > > are > > > > > > > interested. > > > > > > > > > > > > > > <snip> > > > > > > > If that is what you are doing then it could be a good idea > > > > > because > > > > > > > the pre-market data contains trade corrections from > > previous > > > > > days > > > > > > > and I found I had to delete all of that because it wasn't > > > > > palatable > > > > > > to > > > > > > > AB's import method. > > > > > > > </snip> > > > > > > > > > > > > > > I clean all this before import into AB. > > > > > > > > > > > > > > <snip> > > > > > > > Have you successfully imported any of the data into AB in > > > the > > > > > > tick > > > > > > > format? > > > > > > > </snip> > > > > > > > > > > > > > > Yes, no problems with that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is solved for me - I will just ignore out of hours > > > volume. > > > > > > > > > > > > > > Thanks again Brian. > > > > > > > > > > > > > > Best Regards > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > >
