Well darn it. Had a detailed response ready to go and found that GG does not save drafts. POS. Movin on.
The upshot was I tried predicting tick data (see attachment) with a model pretty much identical to the one given here with little real success. It would only resolve if I removed all of the discontinuities in the data - that came to over 20% of the data) The resulting "fantasy" set failed to predict trends that were not in the set with (15% success rate maximum) The conclusion I came to is that actual tick data (which is aggregate of all trading) is not predictable with the given model. NNs (certainly backprop) just try to spline a curve through a training set of input/ouput exemplars. Tick data simply does not seem to provide such a curve. Equivalent to stating what is obvious - stock markets are unpredictable from moment to moment). However one approach that occurred to me was rather than using aggregate tick data, rather use historical data from a given single brokerage that is known to use automated trading. Rather than trying to learn a dog-breakfast of influences given by aggregate data, non-aggregated direct data coming direct from a automated trading algo should prove to be a likely subject for machine-learning. However, if John Nash was correct then these systems would deliberately make bad trades in order to be less predictable. Can't do too much of that though :) There are plenty of examples in Java of backprop. It is well known, and since you are using a known library I doubt there are errors in it, other than sub-optimal coding. Historical FIX data (non-aggregated direct trading transactions) is generally not published anywhere. The model being explored here would be very useful if it were. Cheers! Dominic Cerisano On Sunday, August 5, 2012 2:27:35 PM UTC-4, frye wrote: > > Hey all, > > This post is a fork of a thread in the post "community interest in > machine > learning<https://groups.google.com/forum/?fromgroups#!topic/clojure/heBrnBuUGqs>". > > Some of us were starting to take a deep dive into > clojure-encog<https://github.com/jimpil/clojure-encog> and > I thought it would be a good idea to have a new thread for that. > > So I took a look at the way > encog-java<https://github.com/encog/encog-java-core> > (what clojure-encog <https://github.com/jimpil/clojure-encog> wraps) > loads tick data into it's system. There is a > YahooFinanceLoader<https://github.com/encog/encog-java-core/blob/master/src/main/java/org/encog/ml/data/market/loader/YahooFinanceLoader.java>that > pulls csv data from a URL. But it assumes that prices only have a > daily granularity. Now, the encog-java system seems to have the concept of > granularity going down to the second (see > here<https://github.com/encog/encog-java-core/blob/master/src/main/java/org/encog/util/time/TimeUnit.java>). > But > all of it's market loaders and list of ticks, seem to stop at a time > granularity of daily. See the LoadedMarketData > source<https://github.com/encog/encog-java-core/blob/master/src/main/java/org/encog/ml/data/market/loader/LoadedMarketData.java>, > > which uses a daily-biased > MarketDataType<https://github.com/encog/encog-java-core/blob/master/src/main/java/org/encog/ml/data/market/MarketDataType.java>. > Obviously, > that's not enough if we want to calculate on a second or sub-second > interval. Ultimately the YahooFinanceLoader will give us a list of > LoadedMarketData, which assumes daily price ticks. > > What I need to know is can I give the encog neural net a list of tick data > that has second or sub-second intervals? Back over the clojure-encog, the > thing that normalizes input data, the > make-data<https://github.com/jimpil/clojure-encog/blob/master/src/clojure_encog/training.clj#L41>function > only deals with doubles (not a list of tick data entries). The make-trainer > and > train<https://github.com/jimpil/clojure-encog/blob/master/src/clojure_encog/training.clj#L137>functions > seem to iterate for the number of strategies that you've > specified. But I can't see in > BackPropogation<https://github.com/encog/encog-java-core/blob/master/src/main/java/org/encog/neural/networks/training/propagation/back/Backpropagation.java>or > it's > superclass<https://github.com/encog/encog-java-core/blob/master/src/main/java/org/encog/neural/networks/training/propagation/Propagation.java>, > > where that tick data is actually processed (init and iteration methods seem > to just setup a background process). So I'm left wondering how I can give > the core encog neural-net a list of tick data that has a second or > sub-second granularity? > > > Hmmm > > Tim Washington > Interruptsoftware.ca > 416.843.9060 > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
