Hi Michael, thanks... some more background info on the queries: * note I am using neo ver 2.2 (I guess I should finally upgrade to 3+) * everything I do is via the java api * the queries are traversals and expansions: --- I walk the graph node to node selecting each node by a function of the weight vectors --- I expand around a node to a depth on n for both incoming and outgoing directions --- I commonly use shortest path using dijkstra with my own cost evaluators that use the weight vectors --- once I have a reliable way to write all the properties I will use the graph exclusively in 'read-only' mode. I only write the properties as part of a graph creation process which is a single event usage - fast and predictable creation of course is nice to achieve.
I turn of transaction logging with: keep_logical_logs=false. Let me try using an integer array as a single property and see how that performs. Thanks, John. On Tuesday, August 9, 2016 at 3:35:50 AM UTC-7, Michael Hunger wrote: > > Hi John, > > which kind of "transaction logging did you turn off" ? > > Would you be able to share the queries you are using? > > each double property takes 8 bytes of storage in the property-record > (which are linked in a chain, each property-record can hold up to 4 > 4-byte-storage properties). > > But arrays are optimized, esp. if you have small values in your weights it > tries to use only the significant bits to encode values in an array (but I > think it might only do that for integer values). > > Would you be able to run a test where instead of having 5-10 individual > properties you just use an array with that many entries? > > And perhaps even better project the the floating point values to integer > values in that array. > > I also ask our kernel engineers for other tips in this regard. > > HTH, > > Michael > > On Mon, Aug 8, 2016 at 6:56 PM, John Fry <[email protected] <javascript:> > > wrote: > >> Hello Michael, >> >> the graph is used as follows: >> >> - ~10M nodes; ~200M relationships >> - Each relationship requires multiple floating properties that can be >> considered connecting strength weights. These multiple weights make up a >> weight vector - upto ~20 weights per vector >> - The weights on the relationship are static (or at least they rarely >> change) >> - The weight vector is used to compute custom (very algorithmic in >> nature) costs per link to drive node-to-node traversals, expansions and >> to >> find cost based n-shortest paths >> - The costs per link are calculated in as close to real time as >> possible and are always different and are never stored or written back to >> the relationships in the graph >> >> Regards, John. >> >> On Monday, August 8, 2016 at 12:12:13 AM UTC-7, Michael Hunger wrote: >>> >>> Hi John, >>> >>> Do you have more details on the properties that you add as well as your >>> graph model and queries? Without these details it will be hard to help. >>> >>> It sounds a bit as if your property heavy relationships might be nodes >>> in hiding. >>> >>> Cheers Michael >>> >>> >>> Von meinem iPhone gesendet >>> >>> Am 08.08.2016 um 06:05 schrieb John Fry <[email protected]>: >>> >>> Hi All, >>> >>> In ne04j 2.3 what / where are the limits when storing properties on >>> relationships? >>> >>> I have a graph with about 200M relationships and for each relationship I >>> want to add floating point attributes as properties. >>> Here is what I am experiencing: >>> >>> - adding 2 properties per rel - all works fine; very good performance >>> - adding 5 properties per rel - start to see exceptions/crashes - >>> can be fixed by turning off transaction logging - good performance >>> - adding ~7 properties per rel - performance dramatically fades >>> (10x slower) - occasional exceptions/crashes >>> - adding ~10 properties per real - performance stalls/stops - >>> eventually will crash >>> >>> What is a realistic set of expectations for storing this many properties >>> where the relationship store could easily exceed > 20GB? >>> >>> Regards and thanks for any advice, John. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
