Hi Andrew, I'll let one of the committers answer to your specific data file question. However, you might find some inspiration in this open source demo that some of the Geode team presented at OSCON earlier this year: http://pivotal-open-source-hub.github.io/StockInference-Spark/
This was based on a pre-release version of Geode, so you'll want to sub the M1 release in and see if any other tweaks are required at that point. I believe this video and presentation go with the Github project: http://www.infoq.com/presentations/r-gemfire-spring-xd On Thu, Feb 18, 2016 at 8:58 PM, Andrew Munn <[email protected]> wrote: > What would be the best way to use Geode (or GF) to store and utilize > financial time series data like a stream of stock trades? I have ASCII > files with timestamps that include microseconds: > > 2016-02-17 18:00:00.000660,1926.75,5,5,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,80,85,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,1,86,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,6,92,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,27,119,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,3,122,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,5,127,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,4,131,1926.75,1926.75,14644971,C,43,01, > 2016-02-17 18:00:00.000660,1926.75,2,133,1926.75,1926.75,14644971,C,43,01, > > I have one file per day and each file can have over 1,000,000 rows. My > thought is to fault in the files and parse the ASCII as needed. I know I > could store the data as binary primitives in a file on disk instead of > ASCII for a bit more speed. > > I don't have a cluster of machines to create an HDFS cluster with. My > machine does have 128GB of RAM though. > > Thanks! > -- Greg Chase Global Head, Big Data Communities http://www.pivotal.io/big-data Pivotal Software http://www.pivotal.io/ 650-215-0477 @GregChase Blog: http://geekmarketing.biz/
