Hi Massimo, I just want to understand your use case.
You have a stream of records (log rows in your case) coming in. You process each record, somehow mutating the graph. Then you want to remember that you've already processed that record. If the same record arrives at some later point, you want to know that it has already been processed. If this is an accurate description, I'd like to know what kind of processing and mutating of the graph it is that you do. Maybe you could describe it? Thanks, David On Tue, Feb 15, 2011 at 1:56 PM, Massimo Lusetti <[email protected]> wrote: > On Tue, Feb 15, 2011 at 9:14 PM, Mattias Persson > <[email protected]> wrote: > > > 100 million sounds strange :) but to have a hand full of key/value pairs > > pointing to the same entity is rather normal. Could you elaborate more on > > that use case to let us know why you apparently have super many of those > to > > the same entity? > > I need to elaborate a series of data coming in in a form of a log row > inside a file. The data consist of action taken from users. > > The problem is very similar to having to parse squid log files which > contains authenticated users. Each log file could arrive from more > then one location (squid peering simile) and I've to be sure to not > elaborate the data more then one time. > > So I though I could calculate an hash of each row and index it while > storing the row's Node in the db. After a couple of tests it has been > revealed that storing each row in the db is not feasible since it > would occupy more then 700G of disk space since each node would > consist of at least 800 bytes (node + properties) and I've 222 million > nodes growing. > Since what I really want to know is the existence of an hash within > the index, which means I've already elaborated the unit of work (row), > I though I could simply index each hash to the same node. > > I know it sounds strange cause it's the same to me but I cannot afford > to store that amount of data, even more cause it doesn't contain the > elaborated data yet. > > Any hints is really appreciated. > -- > Massimo > http://meridio.blogspot.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- David Montag Neo Technology, www.neotechnology.com Cell: 650.556.4411 [email protected] _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
