Oh sorry, I might have misunderstood you. Do you see the performance issue when creating the data or when accessing it?
Could you share your graph-creation code? M On Tue, Aug 9, 2016 at 3:51 PM, John Fry <[email protected]> wrote: > Hi Michael, thanks... > > some more background info on the queries: > * note I am using neo ver 2.2 (I guess I should finally upgrade to 3+) > * everything I do is via the java api > * the queries are traversals and expansions: > --- I walk the graph node to node selecting each node by a function of the > weight vectors > --- I expand around a node to a depth on n for both incoming and outgoing > directions > --- I commonly use shortest path using dijkstra with my own cost > evaluators that use the weight vectors > --- once I have a reliable way to write all the properties I will use the > graph exclusively in 'read-only' mode. I only write the properties as part > of a graph creation process which is a single event usage - fast and > predictable creation of course is nice to achieve. > > I turn of transaction logging with: keep_logical_logs=false. > > Let me try using an integer array as a single property and see how that > performs. > > Thanks, John. > > > On Tuesday, August 9, 2016 at 3:35:50 AM UTC-7, Michael Hunger wrote: >> >> Hi John, >> >> which kind of "transaction logging did you turn off" ? >> >> Would you be able to share the queries you are using? >> >> each double property takes 8 bytes of storage in the property-record >> (which are linked in a chain, each property-record can hold up to 4 >> 4-byte-storage properties). >> >> But arrays are optimized, esp. if you have small values in your weights >> it tries to use only the significant bits to encode values in an array (but >> I think it might only do that for integer values). >> >> Would you be able to run a test where instead of having 5-10 individual >> properties you just use an array with that many entries? >> >> And perhaps even better project the the floating point values to integer >> values in that array. >> >> I also ask our kernel engineers for other tips in this regard. >> >> HTH, >> >> Michael >> >> On Mon, Aug 8, 2016 at 6:56 PM, John Fry <[email protected]> wrote: >> >>> Hello Michael, >>> >>> the graph is used as follows: >>> >>> - ~10M nodes; ~200M relationships >>> - Each relationship requires multiple floating properties that can >>> be considered connecting strength weights. These multiple weights make >>> up a >>> weight vector - upto ~20 weights per vector >>> - The weights on the relationship are static (or at least they >>> rarely change) >>> - The weight vector is used to compute custom (very algorithmic in >>> nature) costs per link to drive node-to-node traversals, expansions and >>> to >>> find cost based n-shortest paths >>> - The costs per link are calculated in as close to real time as >>> possible and are always different and are never stored or written back to >>> the relationships in the graph >>> >>> Regards, John. >>> >>> On Monday, August 8, 2016 at 12:12:13 AM UTC-7, Michael Hunger wrote: >>>> >>>> Hi John, >>>> >>>> Do you have more details on the properties that you add as well as your >>>> graph model and queries? Without these details it will be hard to help. >>>> >>>> It sounds a bit as if your property heavy relationships might be nodes >>>> in hiding. >>>> >>>> Cheers Michael >>>> >>>> >>>> Von meinem iPhone gesendet >>>> >>>> Am 08.08.2016 um 06:05 schrieb John Fry <[email protected]>: >>>> >>>> Hi All, >>>> >>>> In ne04j 2.3 what / where are the limits when storing properties on >>>> relationships? >>>> >>>> I have a graph with about 200M relationships and for each relationship >>>> I want to add floating point attributes as properties. >>>> Here is what I am experiencing: >>>> >>>> - adding 2 properties per rel - all works fine; very good >>>> performance >>>> - adding 5 properties per rel - start to see exceptions/crashes - >>>> can be fixed by turning off transaction logging - good performance >>>> - adding ~7 properties per rel - performance dramatically fades >>>> (10x slower) - occasional exceptions/crashes >>>> - adding ~10 properties per real - performance stalls/stops - >>>> eventually will crash >>>> >>>> What is a realistic set of expectations for storing this many >>>> properties where the relationship store could easily exceed > 20GB? >>>> >>>> Regards and thanks for any advice, John. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Neo4j" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
