Hi Micheal,
just to following up on this using the an array of float solved the
performance issue.
However it opens up another problem.
public class ScoredLink {
long id;
float[] lcm = new float[5];
ScoredLink sl = scoredLinks.get(i);
Relationship l = db.getRelationshipById(sl.id);
l.setProperty("lwa_lcm", sl.lcm); //all 16 lcm vals
I can now write 16 floats, as a float[] property attached to each of the
200M relationships here is the problem:
- if float[] contains 0s i can open the db in neo4j-shell and see that
the relationships have the array of 0s
- if float[] contains random
Thanks,
On Tuesday, August 9, 2016 at 1:32:21 PM UTC-7, Michael Hunger wrote:
>
> Oh sorry, I might have misunderstood you.
>
> Do you see the performance issue when creating the data or when accessing
> it?
>
> Could you share your graph-creation code?
>
> M
>
> On Tue, Aug 9, 2016 at 3:51 PM, John Fry <[email protected] <javascript:>
> > wrote:
>
>> Hi Michael, thanks...
>>
>> some more background info on the queries:
>> * note I am using neo ver 2.2 (I guess I should finally upgrade to 3+)
>> * everything I do is via the java api
>> * the queries are traversals and expansions:
>> --- I walk the graph node to node selecting each node by a function of
>> the weight vectors
>> --- I expand around a node to a depth on n for both incoming and outgoing
>> directions
>> --- I commonly use shortest path using dijkstra with my own cost
>> evaluators that use the weight vectors
>> --- once I have a reliable way to write all the properties I will use the
>> graph exclusively in 'read-only' mode. I only write the properties as part
>> of a graph creation process which is a single event usage - fast and
>> predictable creation of course is nice to achieve.
>>
>> I turn of transaction logging with: keep_logical_logs=false.
>>
>> Let me try using an integer array as a single property and see how that
>> performs.
>>
>> Thanks, John.
>>
>>
>> On Tuesday, August 9, 2016 at 3:35:50 AM UTC-7, Michael Hunger wrote:
>>>
>>> Hi John,
>>>
>>> which kind of "transaction logging did you turn off" ?
>>>
>>> Would you be able to share the queries you are using?
>>>
>>> each double property takes 8 bytes of storage in the property-record
>>> (which are linked in a chain, each property-record can hold up to 4
>>> 4-byte-storage properties).
>>>
>>> But arrays are optimized, esp. if you have small values in your weights
>>> it tries to use only the significant bits to encode values in an array (but
>>> I think it might only do that for integer values).
>>>
>>> Would you be able to run a test where instead of having 5-10 individual
>>> properties you just use an array with that many entries?
>>>
>>> And perhaps even better project the the floating point values to integer
>>> values in that array.
>>>
>>> I also ask our kernel engineers for other tips in this regard.
>>>
>>> HTH,
>>>
>>> Michael
>>>
>>> On Mon, Aug 8, 2016 at 6:56 PM, John Fry <[email protected]> wrote:
>>>
>>>> Hello Michael,
>>>>
>>>> the graph is used as follows:
>>>>
>>>> - ~10M nodes; ~200M relationships
>>>> - Each relationship requires multiple floating properties that can
>>>> be considered connecting strength weights. These multiple weights make
>>>> up a
>>>> weight vector - upto ~20 weights per vector
>>>> - The weights on the relationship are static (or at least they
>>>> rarely change)
>>>> - The weight vector is used to compute custom (very algorithmic in
>>>> nature) costs per link to drive node-to-node traversals, expansions and
>>>> to
>>>> find cost based n-shortest paths
>>>> - The costs per link are calculated in as close to real time as
>>>> possible and are always different and are never stored or written back
>>>> to
>>>> the relationships in the graph
>>>>
>>>> Regards, John.
>>>>
>>>> On Monday, August 8, 2016 at 12:12:13 AM UTC-7, Michael Hunger wrote:
>>>>>
>>>>> Hi John,
>>>>>
>>>>> Do you have more details on the properties that you add as well as
>>>>> your graph model and queries? Without these details it will be hard to
>>>>> help.
>>>>>
>>>>> It sounds a bit as if your property heavy relationships might be nodes
>>>>> in hiding.
>>>>>
>>>>> Cheers Michael
>>>>>
>>>>>
>>>>> Von meinem iPhone gesendet
>>>>>
>>>>> Am 08.08.2016 um 06:05 schrieb John Fry <[email protected]>:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> In ne04j 2.3 what / where are the limits when storing properties on
>>>>> relationships?
>>>>>
>>>>> I have a graph with about 200M relationships and for each relationship
>>>>> I want to add floating point attributes as properties.
>>>>> Here is what I am experiencing:
>>>>>
>>>>> - adding 2 properties per rel - all works fine; very good
>>>>> performance
>>>>> - adding 5 properties per rel - start to see exceptions/crashes -
>>>>> can be fixed by turning off transaction logging - good performance
>>>>> - adding ~7 properties per rel - performance dramatically fades
>>>>> (10x slower) - occasional exceptions/crashes
>>>>> - adding ~10 properties per real - performance stalls/stops -
>>>>> eventually will crash
>>>>>
>>>>> What is a realistic set of expectations for storing this many
>>>>> properties where the relationship store could easily exceed > 20GB?
>>>>>
>>>>> Regards and thanks for any advice, John.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Neo4j" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Neo4j" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
