MBs per minute should not be a problem for a modern HDD. See http://www.eecs.berkeley.edu/~rcs/research/interactive_latency.html
Disk seeks are expensive though, so you'll want to buffer your I/O. Big sequential writes are relatively cheap. Log4j's RandomAccessFileAppender (optionally async to ensure bursts don't slow down your app) should be sufficient. For your data structure you could consider fixed length binary records. This is blazingly fast. In a low-latency trading app I was logging ~100MB/sec (slightly overdoing it, ahem). This wrote fixed length binary records to a memory mapped file. Worked very well. Sent from my iPhone > On 2015/06/03, at 17:21, Gary Gregory <[email protected]> wrote: > > I have a use case where I want an appender to write to local storage (a > file), the file should be structured, such that I can query it later (or load > it into a database). Since I will log a lot of data very fast (possibly MBs a > minute), I might or might not use log4j in async mode. But the bottom line is > that I'd like to log to local structured storage without paying the cost of > going through a database layer (NoSQL, JDBC) or a socket. This makes me > wonder if I should create a CSV file appender... which would be easy enough. > > Does anyone here have experience with a use case like this? I'm not crazy > about running MongoDB on the side just to gather logging, but maybe that's > what I'll need to do... > > Thoughts? > > Gary > > -- > E-Mail: [email protected] | [email protected] > Java Persistence with Hibernate, Second Edition > JUnit in Action, Second Edition > Spring Batch in Action > Blog: http://garygregory.wordpress.com > Home: http://garygregory.com/ > Tweet! http://twitter.com/GaryGregory
