On Wed, Feb 16, 2011 at 2:46 AM, David Montag <[email protected]> wrote:
> Hi Massimo, > > I just want to understand your use case. > > You have a stream of records (log rows in your case) coming in. You process > each record, somehow mutating the graph. Then you want to remember that > you've already processed that record. If the same record arrives at some > later point, you want to know that it has already been processed. > > If this is an accurate description, I'd like to know what kind of processing > and mutating of the graph it is that you do. Maybe you could describe it? > Firs of all thanks for your interest. Data contains info about the user who generated the log, the timestamp, the ip and the action taken. The action is composed in a URL-like form, you really can think about it as it would be a squid log row: 172.20.1.6 group1 user1 [29/Jan/2011:02:52:54 +0100] "TRACE http://safebrowsing.clients.mystation.com/safebrowsing/downloads? HTTP/1.1" 403 4534 TCP_DENIED:NONE The processing of the above rule would modify an aggregated data (statistics) for users, groups, ip (network which means location) and actions (URL) Plus on the action (URL) I do some semantics analysis trying to understand what the user is trying to accomplish. The action is coming from a custom application and has a big and wide but limited cases. Last year I got more then 222 millions of rows and this year they're growing so I simply decided to not store the URL in the neo4j DB since is simply cannot afford it, at least in my tests. I would be happy to be contradicted. I really like to play/work with neo4j but I need to find a way to workaround these issues... db dir size and performance. Thanks again for any clue you can give me. -- Massimo http://meridio.blogspot.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
