Hi, Andrey. I was using 2.2.10 but just to be sure, I ran it a second time making sure that 2.2.10 was the first thing in my classpath and I am afraid that I saw it again. It's quite predictable (anywhere between 200 and 250 million edges).
Regards, Phillip On Monday, September 26, 2016 at 9:06:44 AM UTC+1, Andrey Lomakin wrote: > > Hi, > I have looked at your thread dump we have already identified and fixed > your issue in 2.2.9 version. > So if you use 2.2.10 (latest one), you will not experience this problem. > > I strongly recommend using 2.2.10 version because several deadlocks are > fixed in 2.2.9 version also 2.2.10 contains few minor optimizations. > > On Fri, Sep 23, 2016 at 6:51 PM Phillip Henry <phill...@gmail.com > <javascript:>> wrote: > >> Hi, Luca. >> >> > How many GB? >> >> The input file is 22gb of text. >> >> > If the file is ordered ... >> >> You are only sorting by the first account. The second account can be >> anywhere in the entire range. My understanding is that both vertices are >> updated when an edge is written. If this is true, will there not be >> potential contention when the "to" vertex is updated? >> >> > OGraphBatchInsert ... keeps everything in RAM before flushing >> >> I assume I will still have to write retry code in the event of a >> collision (see above)? >> >> > You cna use support --at- orientdb.com ... >> >> Sent. >> >> Regards, >> >> Phill >> >> On Friday, September 23, 2016 at 4:06:49 PM UTC+1, l.garulli wrote: >> >>> On 23 September 2016 at 03:50, Phillip Henry <phill...@gmail.com> wrote: >>> >>>> > How big is your file the sort cannot write? >>>> >>>> One bil-ee-on lines... :-P >>>> >>> >>> How many GB? >>> >>> >>>> > ...This should help a lot. >>>> >>>> The trouble is that the size of a block of contiguous accounts in the >>>> real data is not-uniform (even if it might be with my test data). >>>> Therefore, it is highly likely a contiguous block of account numbers will >>>> span 2 or more batches. This will lead to a lot of contention. In your >>>> example, if Account 2 spills over into the next batch, chances are I'll >>>> have to rollback that batch. >>>> >>>> Don't you also have a problem that if X, Y, Z and W in your example are >>>> account numbers in the next batch, you'll also get contention? Admittedly, >>>> randomization doesn't solve this problem either. >>>> >>> >>> If the file is ordered, you could have X threads (where X is the number >>> of cores) that parse the file not sequentially. For example with 4 threads, >>> you could start the parsing in this way: >>> >>> Thread 1, starts from 0 >>> Thread 2, starts from length * 1/4 >>> Thread 3, starts from length * 2/4 >>> Thread 1, starts from length * 3/4 >>> >>> Of course the parsing should browse until the next LF+LR if it's a CSV. >>> It requires some lines of code, but you could avoid many conflicts. >>> >>> >>>> > you can use the special Batch Importer: OGraphBatchInsert >>>> >>>> Would this not be subject to the same contention problems? >>>> At what point is it flushed to disk? (Obviously, it can't live in heap >>>> forever). >>>> >>> >>> It keeps everything in RAM before flushing. Up to a few hundreds of >>> millions of vertices/edges should be fine if you have a lot of heap, like >>> 58GB (and 4GB of DISKCACHE). It depends by the number of attributes you >>> have. >>> >>> >>>> > You should definitely using transactions with batch size of 100 >>>> items. >>>> >>>> I thought I read somewhere else (can't find the link at the moment) >>>> that you said only use transactions when using the remote protocol? >>>> >>> >>> This was true before v2.2. With v2.2 the management of the transaction >>> is parallel and very light. Transactions work well with graphs because >>> every addEdge() operation is 2 update and having a TX that works like a >>> batch really helps. >>> >>> >>>> >>>> > Please use last 2.2.10. ... try to define 50GB of DISKCACHE and 14GB >>>> of Heap >>>> >>>> Will do on the next run. >>>> >>>> > If happens again, could you please send a thread dump? >>>> >>>> I have the full thread dump but it's on my work machine so can't post >>>> it in this forum (all access to Google Groups is banned by the bank so I >>>> am >>>> writing this on my personal computer). Happy to email them to you. Which >>>> email shall I use? >>>> >>> >>> You cna use support --at- orientdb.com referring at this thread in the >>> subject. >>> >>> >>>> >>>> Phill >>>> >>> >>> >>> Best Regards, >>> >>> Luca Garulli >>> Founder & CEO >>> OrientDB LTD <http://orientdb.com/> >>> >>> Want to share your opinion about OrientDB? >>> Rate & review us at Gartner's Software Review >>> <https://www.gartner.com/reviews/survey/home> >>> >>> >>> >>>> On Friday, September 23, 2016 at 7:41:29 AM UTC+1, l.garulli wrote: >>>> >>>>> On 23 September 2016 at 00:49, Phillip Henry <phill...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, Luca. >>>>>> >>>>> >>>>> Hi Phillip. >>>>> >>>>> >>>>>> I have: >>>>>> >>>>>> 4. sorting is an overhead, albeit outside of Orient. Using the Unix >>>>>> sort command failed with "No space left on device". Oops. OK, so I ran >>>>>> my >>>>>> program to generate the data again, this time it is ordered by the first >>>>>> account number. Performance was much slower as there appeared to be a >>>>>> lot >>>>>> of contention for this account (ie, all writes were contending for this >>>>>> account, even if the other account had less contention). More randomized >>>>>> data was faster. >>>>>> >>>>> >>>>> How big is your file the sort cannot write? Anyway, if you have the >>>>> accounts sorted, you should have transactions of about 100 items where >>>>> the >>>>> bank account and edges are in the same transaction. This should help a >>>>> lot. >>>>> Example: >>>>> >>>>> Account 1 -> Payment 1 -> Account X >>>>> Account 1 -> Payment 2 -> Account Y >>>>> Account 1 -> Payment 3 -> Account Z >>>>> Account 2 -> Payment 1 -> Account X >>>>> Account 2 -> Payment 1 -> Account W >>>>> >>>>> If the transaction batch is 5 (I suggest you to start with 100), all >>>>> the operations are executed in one transaction. In another thread has: >>>>> >>>>> Account 99 -> Payment 1 -> Account W >>>>> >>>>> It could go in conflict because the shared Account W. >>>>> >>>>> If you can export Account's IDs that are numbers and incremental, you >>>>> can use the special Batch Importer: OGraphBatchInsert. Example: >>>>> >>>>> OGraphBatchInsert batch = new OGraphBatchInsert("plocal:/temp/mydb", >>>>> "admin", "admin"); >>>>> batch.begin(); >>>>> >>>>> batch.createEdge(0L, 1L, null); // CREATE EDGES BETWEEN VERTEX 0 and 1. >>>>> IF VERTICES >>>>> >>>>> // DON'T EXISTS, ARE CREATED IMPLICITELY >>>>> batch.createEdge(1L, 2L, null); >>>>> batch.createEdge(2L, 0L, null); >>>>> >>>>> >>>>> batch.createVertex(3L); // CREATE AN NON CONNECTED VERTEX >>>>> >>>>> >>>>> Map<String, Object> vertexProps = new HashMap<String, Object>(); >>>>> vertexProps.put("foo", "foo"); >>>>> vertexProps.put("bar", 3); >>>>> batch.setVertexProperties(0L, vertexProps); // SET PROPERTY FOR VERTEX 0 >>>>> batch.end(); >>>>> >>>>> This is blazing fast, but uses Heap so run it with a lot of it. >>>>> >>>>> >>>>>> >>>>>> 6. I've mutlithreaded my loader. The details are now: >>>>>> >>>>>> - using plocal >>>>>> - using 30 threads >>>>>> - not using transactions (OrientGraphFactory.getNoTx) >>>>>> >>>>> >>>>> You should definitely using transactions with batch size of 100 items. >>>>> This speeds up things. >>>>> >>>>> >>>>>> - retrying forever upon write collisions. >>>>>> - using Orient 2.2.7. >>>>>> >>>>> >>>>> Please use last 2.2.10. >>>>> >>>>> >>>>>> - using -XX:MaxDirectMemorySize:258040m >>>>>> >>>>> >>>>> This is not really important, it's just an upper bound for the JVM. >>>>> Please set it to 512GB so you can forget about it. The 2 most important >>>>> values are DISKCACHE and JVM heap. The sum must lower than the available >>>>> RAM in the server before you run OrientDB. >>>>> >>>>> If you have 64GB, try to define 50GB of DISKCACHE and 14GB of Heap. >>>>> >>>>> If you use the Batch Importer, you should use more Heap and less >>>>> DISKCACHE. >>>>> >>>>> >>>>>> The good news is I've achieved an initial write throughput of about >>>>>> 30k/second. >>>>>> >>>>>> The bad news is I've tried several runs and only been able to achieve >>>>>> 200mil < number of writes < 300mil. >>>>>> >>>>>> The first time I tried it, the loader deadlocked. Using jstat showed >>>>>> that the deadlock was between 3 threads at: >>>>>> - >>>>>> OOneKeyEntryPerKeyLockManager.acquireLock(OOneKeyEntryPerKeyLockManager.java:173) >>>>>> - >>>>>> OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:210) >>>>>> - >>>>>> OOneKeyEntryPerKeyLockManager.acquireLock(OOneKeyEntryPerKeyLockManager.java:171) >>>>>> >>>>> >>>>> If happens again, could you please send a thread dump? >>>>> >>>>> >>>>>> The second time it failed was due to a NullPointerException at >>>>>> OByteBufferPool.java:297. I've looked at the code and the only way I can >>>>>> see this happening is if OByteBufferPool.allocateBuffer throws an error >>>>>> (perhaps an OutOfMemoryError in java.nio.Bits.reserveMemory). This >>>>>> StackOverflow posting ( >>>>>> http://stackoverflow.com/questions/8462200/examples-of-forcing-freeing-of-native-memory-direct-bytebuffer-has-allocated-us) >>>>>> >>>>>> seems to indicate that this can happen if the underlying >>>>>> DirectByteBuffer's >>>>>> Cleaner doesn't have its clean() method called. >>>>>> >>>>> >>>>> This is because the database was bigger than this setting: - using >>>>> -XX:MaxDirectMemorySize:258040m. Please set this at 512GB (see above). >>>>> >>>>> >>>>>> Alternatively, I followed the SO suggestion and lowered the heap >>>>>> space to a mere 1gb (it was 50gb) to make the GC more active. >>>>>> Unfortunately, after a good start, the job is still running some 15 >>>>>> hours >>>>>> later with a hugely reduced write throughput (~ 7k/s). Jstat shows 4292 >>>>>> full GCs taking a total time of 4597s - not great but not hugely awful >>>>>> either. At this rate, the remaining 700mil or so payments are going to >>>>>> take >>>>>> another 30 hours. >>>>>> >>>>> >>>>> See above the suggested settings. >>>>> >>>>> >>>>>> 7. Even with the highest throughput I have achieved, 30k writes per >>>>>> second, I'm looking at about 20 hours of loading. We've taken the same >>>>>> data >>>>>> and, after trial and error that was not without its own problems, put it >>>>>> into Neo4J in 37 minutes. This is a significant difference. It appears >>>>>> that >>>>>> they are approaching the problem differently to avoid contention on >>>>>> updating the vertices during an edge write. >>>>>> >>>>> >>>>> With all this suggestion you should be able to have much better >>>>> numbers. If you can use the Batch Importer the number should be close to >>>>> Neo4j. >>>>> >>>>> >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> Regards, >>>>>> >>>>>> Phillip >>>>>> >>>>>> >>>>> >>>>> Best Regards, >>>>> >>>>> Luca Garulli >>>>> Founder & CEO >>>>> OrientDB LTD <http://orientdb.com/> >>>>> >>>>> Want to share your opinion about OrientDB? >>>>> Rate & review us at Gartner's Software Review >>>>> <https://www.gartner.com/reviews/survey/home> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> On Thursday, September 15, 2016 at 10:06:44 PM UTC+1, l.garulli wrote: >>>>>>> >>>>>>> On 15 September 2016 at 09:54, Phillip Henry <phill...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, Luca. >>>>>>>> >>>>>>> >>>>>>> Hi Phillip, >>>>>>> >>>>>>> 3. Yes, default configuration. Apart from adding an index for >>>>>>>> ACCOUNTS, I did nothing further. >>>>>>>> >>>>>>> >>>>>>> Ok, so you have writeQuorum="majority" that means 2 sycnhronous >>>>>>> writes and 1 asynchronous per transaction. >>>>>>> >>>>>>> >>>>>>>> 4. Good question. With real data, we expect it to be as you >>>>>>>> suggest: some nodes with the majority of the payments (eg, >>>>>>>> supermarkets). >>>>>>>> However, for the test data, payments were assigned randomly and, >>>>>>>> therefore, >>>>>>>> should be uniformly distributed. >>>>>>>> >>>>>>> >>>>>>> What's your average in terms of number of edges? <10, <50, <200, >>>>>>> <1000? >>>>>>> >>>>>>> >>>>>>>> 2. Yes, I tried plocal minutes after posting (d'oh!). I saw a good >>>>>>>> improvement. It started about 3 times faster and got faster still >>>>>>>> (about 10 >>>>>>>> times faster) by the time I checked this morning on a job running >>>>>>>> overnight. However, even though it is now running at about 7k >>>>>>>> transactions >>>>>>>> per second, a billion edges is still going to take about 40 hours. So, >>>>>>>> I >>>>>>>> ask myself: is there anyway I can make it faster still? >>>>>>>> >>>>>>> >>>>>>> Here it's missing the usage of AUTO-SHARDING INDEX. Example: >>>>>>> >>>>>>> accountClass.createIndex("Account.number", >>>>>>> OClass.INDEX_TYPE.UNIQUE.toString(), (OProgressListener) null, >>>>>>> (ODocument) null, >>>>>>> "AUTOSHARDING", new String[] { "number" }); >>>>>>> >>>>>>> In this way you should go more in parallel, because the index is >>>>>>> distributed across all the shards (clusters) of Account class. you >>>>>>> should >>>>>>> have 32 of them by default because you have 32 cores. >>>>>>> >>>>>>> Please let me know if by sorting the from_accounts and with this >>>>>>> change if it's much faster. >>>>>>> >>>>>>> This is the best you can have out of the box. To push numbers up >>>>>>> it's slightly more complicated: you should be sure that transactions go >>>>>>> in >>>>>>> parallel and they aren't serialized. This is possible by playing with >>>>>>> internal OrientDB settings (mainly the distributed workerThreads), by >>>>>>> having many clusters per class (You could try with 128 first and see >>>>>>> how >>>>>>> it's going). >>>>>>> >>>>>>> >>>>>>>> I assume when I start the servers up in distributed mode once more, >>>>>>>> the data will then be distributed across all nodes in the cluster? >>>>>>>> >>>>>>> >>>>>>> That's right. >>>>>>> >>>>>>> >>>>>>>> 3. I'll return to concurrent, remote inserts when this job has >>>>>>>> finished. Hopefully, a smaller batch size will mean there is no >>>>>>>> degradation >>>>>>>> in performance either... FYI: with a somewhat unscientific approach, I >>>>>>>> was >>>>>>>> polling the server JVM with JStack and saw only a single thread doing >>>>>>>> all >>>>>>>> the work and it *seemed* to spend a lot of its time in ODirtyManager >>>>>>>> on >>>>>>>> collection manipulation. >>>>>>>> >>>>>>> >>>>>>> I think it's because you didn't use the AUTO-SHARDING index. >>>>>>> Furthermore running distributed, unfortunately, means the tree ridbag >>>>>>> is >>>>>>> not available (we will support it in the future), so every change to >>>>>>> the >>>>>>> edges takes a lot of CPU to demarshall and marshall the entire edge >>>>>>> list >>>>>>> everytime you update a vertex. That's why my recommendation about >>>>>>> sorting >>>>>>> the vertices. >>>>>>> >>>>>>> >>>>>>>> I totally appreciate that performance tuning is an empirical >>>>>>>> science, but do you have any opinions as to which would probably be >>>>>>>> faster: >>>>>>>> single-threaded plocal or multithreaded remote? >>>>>>>> >>>>>>> >>>>>>> With v2.2 yo can go in parallel, by using the tips above. For sure >>>>>>> the replication has a cost. I'm sure you can go much faster with just >>>>>>> one >>>>>>> node and then start the other 2 nodes to have the database replicated >>>>>>> automatically. At least for the first massive insertion. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Phillip >>>>>>>> >>>>>>> >>>>>>> Luca >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Wednesday, September 14, 2016 at 3:48:56 PM UTC+1, Phillip Henry >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, guys. >>>>>>>>> >>>>>>>>> I'm conducting a proof-of-concept for a large bank (Luca, we had a >>>>>>>>> 'phone conf on August 5...) and I'm trying to bulk insert a humongous >>>>>>>>> amount of data: 1 million vertices and 1 billion edges. >>>>>>>>> >>>>>>>>> Firstly, I'm impressed about how easy it was to configure a >>>>>>>>> cluster. However, the performance of batch inserting is bad (and >>>>>>>>> seems to >>>>>>>>> get considerably worse as I add more data). It starts at about 2k >>>>>>>>> vertices-and-edges per second and deteriorates to about 500/second >>>>>>>>> after >>>>>>>>> only about 3 million edges have been added. This also takes ~ 30 >>>>>>>>> minutes. >>>>>>>>> Needless to say that 1 billion payments (edges) will take over a week >>>>>>>>> at >>>>>>>>> this rate. >>>>>>>>> >>>>>>>>> This is a show-stopper for us. >>>>>>>>> >>>>>>>>> My data model is simply payments between accounts and I store it >>>>>>>>> in one large file. It's just 3 fields and looks like: >>>>>>>>> >>>>>>>>> FROM_ACCOUNT TO_ACCOUNT AMOUNT >>>>>>>>> >>>>>>>>> In the test data I generated, I had 1 million accounts and 1 >>>>>>>>> billion payments randomly distributed between pairs of accounts. >>>>>>>>> >>>>>>>>> I have 2 classes in OrientDB: ACCOUNTS (extending V) and PAYMENT >>>>>>>>> (extending E). There is a UNIQUE_HASH_INDEX on ACCOUNTS for the >>>>>>>>> account >>>>>>>>> number (a string). >>>>>>>>> >>>>>>>>> We're using OrientDB 2.2.7. >>>>>>>>> >>>>>>>>> My batch size is 5k and I am using the "remote" protocol to >>>>>>>>> connect to our cluster. >>>>>>>>> >>>>>>>>> I'm using JDK 8 and my 3 boxes are beefy machines (32 cores each) >>>>>>>>> but without SSDs. I wrote the importing code myself but did nothing >>>>>>>>> 'clever' (I think) and used the Graph API. This client code has been >>>>>>>>> given >>>>>>>>> lots of memory and using jstat I can see it is not excessively GCing. >>>>>>>>> >>>>>>>>> So, my questions are: >>>>>>>>> >>>>>>>>> 1. what kind of performance can I realistically expect and can I >>>>>>>>> improve what I have at the moment? >>>>>>>>> >>>>>>>>> 2. what kind of degradation should I expect as the graph grows? >>>>>>>>> >>>>>>>>> Thanks, guys. >>>>>>>>> >>>>>>>>> Phillip >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "OrientDB" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to orient-databa...@googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "OrientDB" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to orient-databa...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "OrientDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to orient-databa...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to orient-databa...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- > Best regards, > Andrey Lomakin, R&D lead. > OrientDB Ltd > > twitter: @Andrey_Lomakin > linkedin: https://ua.linkedin.com/in/andreylomakin > blogger: http://andreylomakin.blogspot.com/ > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
