Hi, If I understand you correctly, you need to do two things when processing a row:
- Make sure it has never been processed before - Add it to the statistics (which is part of the graph structure) I would suggest the following, most of which I think you already do: - Calculate the hash of the row, and lookup in the index of hashes to make sure it does not already exist - If it is unique, parse the row for the statistics, create the new node and connect it to the relevant statistics nodes in your graph. For example, you might have a tree of known users, and each user is connected to the log lines for events they created, and therefor the number of relations is the counter for that user. If the counters are likely to get too big, you need to divide the stats into finer steps, like one counter per user-month, or user-week, etc. Other stats, like websites visited can be handled similarly. - Do NOT store the actual row in the node you created, since it takes too much space. Perhaps store meta-data about the row, like reference to logfile and line in the file, if any future analysis needs to be re-run and get access to the original data. This should save a lot of space. The above design might help, but I can see a few possible issues: - If you use a lucene index for the hashcode, then that index can get quite big. I'm not sure if this is the case, but if it is, rather use your own statistics tree structures as the uniqueness test also. Especially if you base you tree on a fine grained composite key like user-day, a search on the graph-based index should be very fast. This means you are also combining the uniqueness test and the statistics calculation into one step. - If you have lots of separate statistics trees connected to the data, the relationships file can become a bottleneck. For example, if you build 10 statistics trees, then each log row will have ten relationships, so for 220M rows, you have 2.20billion relationships (or more). This can be a big problem. One way around it is to build one composite statistics tree with all properties combined, so each node has only one relationship into the tree. The risk is that even though you save a huge amount on the first level of relationships, the tree itself is larger. If there are zero correlations between properties, the gains are small. However, in the real world there are correlations. For example, each user only visits a subset of websites, so the number of user-website combinations is much smaller than users * websites, so the combined tree is smaller than maximum theoretical size and you do reduce the total relationship count. If you are interested, I have some initial prototype code building such a composite index at https://github.com/craigtaverner/amanzi-index Cheers, Craig On Wed, Feb 16, 2011 at 4:15 PM, Massimo Lusetti <[email protected]> wrote: > On Wed, Feb 16, 2011 at 2:46 AM, David Montag > <[email protected]> wrote: > > > Hi Massimo, > > > > I just want to understand your use case. > > > > You have a stream of records (log rows in your case) coming in. You > process > > each record, somehow mutating the graph. Then you want to remember that > > you've already processed that record. If the same record arrives at some > > later point, you want to know that it has already been processed. > > > > If this is an accurate description, I'd like to know what kind of > processing > > and mutating of the graph it is that you do. Maybe you could describe it? > > > > Firs of all thanks for your interest. > > Data contains info about the user who generated the log, the > timestamp, the ip and the action taken. > > The action is composed in a URL-like form, you really can think about > it as it would be a squid log row: > 172.20.1.6 group1 user1 [29/Jan/2011:02:52:54 +0100] "TRACE > http://safebrowsing.clients.mystation.com/safebrowsing/downloads? > HTTP/1.1" 403 4534 TCP_DENIED:NONE > > The processing of the above rule would modify an aggregated data > (statistics) for users, groups, ip (network which means location) and > actions (URL) > > Plus on the action (URL) I do some semantics analysis trying to > understand what the user is trying to accomplish. The action is coming > from a custom application and has a big and wide but limited cases. > > Last year I got more then 222 millions of rows and this year they're > growing so I simply decided to not store the URL in the neo4j DB since > is simply cannot afford it, at least in my tests. I would be happy to > be contradicted. > > I really like to play/work with neo4j but I need to find a way to > workaround these issues... db dir size and performance. > > Thanks again for any clue you can give me. > -- > Massimo > http://meridio.blogspot.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
