Hi, Adam, great thanks for your detailed reply, the three videos are very referential for me. Actually, the App submitted to IBM Spark Contest is a very small demo, I'll do much more work to enhance that model, and recently we just started a new project which aims to building a platform that makes it possible and easy for Chinese financial institutions to analysis high-frequency market data, which will be ~30G one day.
great thanks again, On Thu, Sep 1, 2016 at 10:25 PM, Adam Roberts <arobe...@uk.ibm.com> wrote: > Hi, yes, there's definitely a market for Apache Spark and financial > institutions, I can't provide specific details but to answer your survey: > "yes" and "more than a few GB!" > > Here are a couple of examples showing Spark with financial data, full > disclosure that I work for IBM, I'm sure there are lots more examples you > can find too: > > - https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be > used with simple sentiment analysis to figure out correlations between real > world events and stock market changes. The Spark specific part is from 3 04 > until the 8th minute > - https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example > where Spark is again used with sentiment analysis. One could also analyse > financial data to identify trends, I think a lot of the machine learning > APIs will be useful here e.g. logistic regression with many features could > be used to decide whether or not an investment is a good idea based on > training data (so we'd look at real outcomes from previous speculations) > > > In both cases you can see Spark is a very important component for > performing the calculations with financial data. > > I also know that Goldman Sachs mentioned they are interested in Spark, one > talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is more > evidence of the financial industries paying attention to big data and Spark. > > Regarding your app: I expected it to be similar to the first example where > the signals you mention are real world events (e.g. the fed lowers interest > rates or companies are rumoured to either be about to float or be > acquired). > > At the 4 30 part I think you actually identify previous index values and > extrapolate what they are likely to become using, so in theory your system > would become more accurate over time although would going off indexes alone > be sufficient (if indeed this is what you're doing). > > I think you'd want to combine this with real world speculation/news to > figure out *why* the price is likely to change, how much by and in which > direction. > > I agree that Apache Spark can be just the right tool for doing the heavy > lifting required for analysis, computation and modelling of big data so > looking forward to future Spark work in this area, and I wonder how we as > Spark developers can make it easier/more powerful for Spark users to do so > > > > > From: "Taotao.Li" <charles.up...@gmail.com> > To: user <user@spark.apache.org> > Date: 30/08/2016 14:14 > Subject: Hi, guys, does anyone use Spark in finance market? > ------------------------------ > > > > > Hi, guys, > > I'm a quant engineer in China, and I believe it's very promising when > using Spark in the financial market. But I didn't find cases which combine > spark and finance. > > So here I wanna do a small survey: > > - do you guys use Spark in financial market related project? > - if yes, how large data was fed in your spark application? > > > thanks a lot. > > *___________________*** > A little ad, I attended IBM Spark Hackathon, which is here : > *http://apachespark.devpost.com/* <http://apachespark.devpost.com/> , and > I submitted a small application, which will be used in my strategies, hope > you guys and give me a vote and some suggestions on how to use spark in > financial market, to discover some trade opportunity. > > here is my small app: > *http://devpost.com/software/spark-in-finance-quantitative-investing* > <http://devpost.com/software/spark-in-finance-quantitative-investing> > > thanks a lot. > > > -- > *___________________* > Quant | Engineer | Boy > *___________________* > *blog*: *http://litaotao.github.io* > <http://litaotao.github.io?utm_source=spark_mail> > *github*: *www.github.com/litaotao* <http://www.github.com/litaotao> > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number > 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > -- *___________________* Quant | Engineer | Boy *___________________* *blog*: http://litaotao.github.io <http://litaotao.github.io?utm_source=spark_mail> *github*: www.github.com/litaotao
