Hi Philip , Could you send thread dump for 2.2.10 version ? пт, 30 Сен 2016, 8:47 Phillip Henry <phillhe...@gmail.com>:
> Hi, Andrey. > > I was using 2.2.10 but just to be sure, I ran it a second time making sure > that 2.2.10 was the first thing in my classpath and I am afraid that I saw > it again. It's quite predictable (anywhere between 200 and 250 million > edges). > > Regards, > > Phillip > > > On Monday, September 26, 2016 at 9:06:44 AM UTC+1, Andrey Lomakin wrote: > >> Hi, >> I have looked at your thread dump we have already identified and fixed >> your issue in 2.2.9 version. >> So if you use 2.2.10 (latest one), you will not experience this problem. >> >> I strongly recommend using 2.2.10 version because several deadlocks are >> fixed in 2.2.9 version also 2.2.10 contains few minor optimizations. >> > >> On Fri, Sep 23, 2016 at 6:51 PM Phillip Henry <phill...@gmail.com> wrote: >> >>> Hi, Luca. >>> >>> > How many GB? >>> >>> The input file is 22gb of text. >>> >>> > If the file is ordered ... >>> >>> You are only sorting by the first account. The second account can be >>> anywhere in the entire range. My understanding is that both vertices are >>> updated when an edge is written. If this is true, will there not be >>> potential contention when the "to" vertex is updated? >>> >>> > OGraphBatchInsert ... keeps everything in RAM before flushing >>> >>> I assume I will still have to write retry code in the event of a >>> collision (see above)? >>> >>> > You cna use support --at- orientdb.com ... >>> >>> Sent. >>> >>> Regards, >>> >>> Phill >>> >>> On Friday, September 23, 2016 at 4:06:49 PM UTC+1, l.garulli wrote: >>> >>>> On 23 September 2016 at 03:50, Phillip Henry <phill...@gmail.com> >>>> wrote: >>>> >>>>> > How big is your file the sort cannot write? >>>>> >>>>> One bil-ee-on lines... :-P >>>>> >>>> >>>> How many GB? >>>> >>>> >>>>> > ...This should help a lot. >>>>> >>>>> The trouble is that the size of a block of contiguous accounts in the >>>>> real data is not-uniform (even if it might be with my test data). >>>>> Therefore, it is highly likely a contiguous block of account numbers will >>>>> span 2 or more batches. This will lead to a lot of contention. In your >>>>> example, if Account 2 spills over into the next batch, chances are I'll >>>>> have to rollback that batch. >>>>> >>>>> Don't you also have a problem that if X, Y, Z and W in your example >>>>> are account numbers in the next batch, you'll also get contention? >>>>> Admittedly, randomization doesn't solve this problem either. >>>>> >>>> >>>> If the file is ordered, you could have X threads (where X is the number >>>> of cores) that parse the file not sequentially. For example with 4 threads, >>>> you could start the parsing in this way: >>>> >>>> Thread 1, starts from 0 >>>> Thread 2, starts from length * 1/4 >>>> Thread 3, starts from length * 2/4 >>>> Thread 1, starts from length * 3/4 >>>> >>>> Of course the parsing should browse until the next LF+LR if it's a CSV. >>>> It requires some lines of code, but you could avoid many conflicts. >>>> >>>> >>>>> > you can use the special Batch Importer: OGraphBatchInsert >>>>> >>>>> Would this not be subject to the same contention problems? >>>>> At what point is it flushed to disk? (Obviously, it can't live in heap >>>>> forever). >>>>> >>>> >>>> It keeps everything in RAM before flushing. Up to a few hundreds of >>>> millions of vertices/edges should be fine if you have a lot of heap, like >>>> 58GB (and 4GB of DISKCACHE). It depends by the number of attributes you >>>> have. >>>> >>>> >>>>> > You should definitely using transactions with batch size of 100 >>>>> items. >>>>> >>>>> I thought I read somewhere else (can't find the link at the moment) >>>>> that you said only use transactions when using the remote protocol? >>>>> >>>> >>>> This was true before v2.2. With v2.2 the management of the transaction >>>> is parallel and very light. Transactions work well with graphs because >>>> every addEdge() operation is 2 update and having a TX that works like a >>>> batch really helps. >>>> >>>> >>>>> >>>>> > Please use last 2.2.10. ... try to define 50GB of DISKCACHE and 14GB >>>>> of Heap >>>>> >>>>> Will do on the next run. >>>>> >>>>> > If happens again, could you please send a thread dump? >>>>> >>>>> I have the full thread dump but it's on my work machine so can't post >>>>> it in this forum (all access to Google Groups is banned by the bank so I >>>>> am >>>>> writing this on my personal computer). Happy to email them to you. Which >>>>> email shall I use? >>>>> >>>> >>>> You cna use support --at- orientdb.com referring at this thread in the >>>> subject. >>>> >>>> >>>>> >>>>> Phill >>>>> >>>> >>>> >>>> Best Regards, >>>> >>>> Luca Garulli >>>> Founder & CEO >>>> OrientDB LTD <http://orientdb.com/> >>>> >>>> Want to share your opinion about OrientDB? >>>> Rate & review us at Gartner's Software Review >>>> <https://www.gartner.com/reviews/survey/home> >>>> >>>> >>>> >>>>> On Friday, September 23, 2016 at 7:41:29 AM UTC+1, l.garulli wrote: >>>>> >>>>>> On 23 September 2016 at 00:49, Phillip Henry <phill...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, Luca. >>>>>>> >>>>>> >>>>>> Hi Phillip. >>>>>> >>>>>> >>>>>>> I have: >>>>>>> >>>>>>> 4. sorting is an overhead, albeit outside of Orient. Using the Unix >>>>>>> sort command failed with "No space left on device". Oops. OK, so I ran >>>>>>> my >>>>>>> program to generate the data again, this time it is ordered by the first >>>>>>> account number. Performance was much slower as there appeared to be a >>>>>>> lot >>>>>>> of contention for this account (ie, all writes were contending for this >>>>>>> account, even if the other account had less contention). More randomized >>>>>>> data was faster. >>>>>>> >>>>>> >>>>>> How big is your file the sort cannot write? Anyway, if you have the >>>>>> accounts sorted, you should have transactions of about 100 items where >>>>>> the >>>>>> bank account and edges are in the same transaction. This should help a >>>>>> lot. >>>>>> Example: >>>>>> >>>>>> Account 1 -> Payment 1 -> Account X >>>>>> Account 1 -> Payment 2 -> Account Y >>>>>> Account 1 -> Payment 3 -> Account Z >>>>>> Account 2 -> Payment 1 -> Account X >>>>>> Account 2 -> Payment 1 -> Account W >>>>>> >>>>>> If the transaction batch is 5 (I suggest you to start with 100), all >>>>>> the operations are executed in one transaction. In another thread has: >>>>>> >>>>>> Account 99 -> Payment 1 -> Account W >>>>>> >>>>>> It could go in conflict because the shared Account W. >>>>>> >>>>>> If you can export Account's IDs that are numbers and incremental, you >>>>>> can use the special Batch Importer: OGraphBatchInsert. Example: >>>>>> >>>>>> OGraphBatchInsert batch = new OGraphBatchInsert("plocal:/temp/mydb", >>>>>> "admin", "admin"); >>>>>> batch.begin(); >>>>>> >>>>>> batch.createEdge(0L, 1L, null); // CREATE EDGES BETWEEN VERTEX 0 and 1. >>>>>> IF VERTICES >>>>>> >>>>>> // DON'T EXISTS, ARE CREATED IMPLICITELY >>>>>> batch.createEdge(1L, 2L, null); >>>>>> batch.createEdge(2L, 0L, null); >>>>>> >>>>>> >>>>>> batch.createVertex(3L); // CREATE AN NON CONNECTED VERTEX >>>>>> >>>>>> >>>>>> Map<String, Object> vertexProps = new HashMap<String, Object>(); >>>>>> vertexProps.put("foo", "foo"); >>>>>> vertexProps.put("bar", 3); >>>>>> batch.setVertexProperties(0L, vertexProps); // SET PROPERTY FOR VERTEX 0 >>>>>> batch.end(); >>>>>> >>>>>> This is blazing fast, but uses Heap so run it with a lot of it. >>>>>> >>>>>> >>>>>>> >>>>>>> 6. I've mutlithreaded my loader. The details are now: >>>>>>> >>>>>>> - using plocal >>>>>>> - using 30 threads >>>>>>> - not using transactions (OrientGraphFactory.getNoTx) >>>>>>> >>>>>> >>>>>> You should definitely using transactions with batch size of 100 >>>>>> items. This speeds up things. >>>>>> >>>>>> >>>>>>> - retrying forever upon write collisions. >>>>>>> - using Orient 2.2.7. >>>>>>> >>>>>> >>>>>> Please use last 2.2.10. >>>>>> >>>>>> >>>>>>> - using -XX:MaxDirectMemorySize:258040m >>>>>>> >>>>>> >>>>>> This is not really important, it's just an upper bound for the JVM. >>>>>> Please set it to 512GB so you can forget about it. The 2 most important >>>>>> values are DISKCACHE and JVM heap. The sum must lower than the available >>>>>> RAM in the server before you run OrientDB. >>>>>> >>>>>> If you have 64GB, try to define 50GB of DISKCACHE and 14GB of Heap. >>>>>> >>>>>> If you use the Batch Importer, you should use more Heap and less >>>>>> DISKCACHE. >>>>>> >>>>>> >>>>>>> The good news is I've achieved an initial write throughput of about >>>>>>> 30k/second. >>>>>>> >>>>>>> The bad news is I've tried several runs and only been able to >>>>>>> achieve 200mil < number of writes < 300mil. >>>>>>> >>>>>>> The first time I tried it, the loader deadlocked. Using jstat showed >>>>>>> that the deadlock was between 3 threads at: >>>>>>> - >>>>>>> OOneKeyEntryPerKeyLockManager.acquireLock(OOneKeyEntryPerKeyLockManager.java:173) >>>>>>> - >>>>>>> OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:210) >>>>>>> - >>>>>>> OOneKeyEntryPerKeyLockManager.acquireLock(OOneKeyEntryPerKeyLockManager.java:171) >>>>>>> >>>>>> >>>>>> If happens again, could you please send a thread dump? >>>>>> >>>>>> >>>>>>> The second time it failed was due to a NullPointerException at >>>>>>> OByteBufferPool.java:297. I've looked at the code and the only way I can >>>>>>> see this happening is if OByteBufferPool.allocateBuffer throws an error >>>>>>> (perhaps an OutOfMemoryError in java.nio.Bits.reserveMemory). This >>>>>>> StackOverflow posting ( >>>>>>> http://stackoverflow.com/questions/8462200/examples-of-forcing-freeing-of-native-memory-direct-bytebuffer-has-allocated-us) >>>>>>> seems to indicate that this can happen if the underlying >>>>>>> DirectByteBuffer's >>>>>>> Cleaner doesn't have its clean() method called. >>>>>>> >>>>>> >>>>>> This is because the database was bigger than this setting: - using >>>>>> -XX:MaxDirectMemorySize:258040m. Please set this at 512GB (see above). >>>>>> >>>>>> >>>>>>> Alternatively, I followed the SO suggestion and lowered the heap >>>>>>> space to a mere 1gb (it was 50gb) to make the GC more active. >>>>>>> Unfortunately, after a good start, the job is still running some 15 >>>>>>> hours >>>>>>> later with a hugely reduced write throughput (~ 7k/s). Jstat shows 4292 >>>>>>> full GCs taking a total time of 4597s - not great but not hugely awful >>>>>>> either. At this rate, the remaining 700mil or so payments are going to >>>>>>> take >>>>>>> another 30 hours. >>>>>>> >>>>>> >>>>>> See above the suggested settings. >>>>>> >>>>>> >>>>>>> 7. Even with the highest throughput I have achieved, 30k writes per >>>>>>> second, I'm looking at about 20 hours of loading. We've taken the same >>>>>>> data >>>>>>> and, after trial and error that was not without its own problems, put it >>>>>>> into Neo4J in 37 minutes. This is a significant difference. It appears >>>>>>> that >>>>>>> they are approaching the problem differently to avoid contention on >>>>>>> updating the vertices during an edge write. >>>>>>> >>>>>> >>>>>> With all this suggestion you should be able to have much better >>>>>> numbers. If you can use the Batch Importer the number should be close to >>>>>> Neo4j. >>>>>> >>>>>> >>>>>>> >>>>>>> Thoughts? >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Phillip >>>>>>> >>>>>>> >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> Luca Garulli >>>>>> Founder & CEO >>>>>> OrientDB LTD <http://orientdb.com/> >>>>>> >>>>>> Want to share your opinion about OrientDB? >>>>>> Rate & review us at Gartner's Software Review >>>>>> <https://www.gartner.com/reviews/survey/home> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> On Thursday, September 15, 2016 at 10:06:44 PM UTC+1, l.garulli >>>>>>> wrote: >>>>>>>> >>>>>>>> On 15 September 2016 at 09:54, Phillip Henry <phill...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, Luca. >>>>>>>>> >>>>>>>> >>>>>>>> Hi Phillip, >>>>>>>> >>>>>>>> 3. Yes, default configuration. Apart from adding an index for >>>>>>>>> ACCOUNTS, I did nothing further. >>>>>>>>> >>>>>>>> >>>>>>>> Ok, so you have writeQuorum="majority" that means 2 sycnhronous >>>>>>>> writes and 1 asynchronous per transaction. >>>>>>>> >>>>>>>> >>>>>>>>> 4. Good question. With real data, we expect it to be as you >>>>>>>>> suggest: some nodes with the majority of the payments (eg, >>>>>>>>> supermarkets). >>>>>>>>> However, for the test data, payments were assigned randomly and, >>>>>>>>> therefore, >>>>>>>>> should be uniformly distributed. >>>>>>>>> >>>>>>>> >>>>>>>> What's your average in terms of number of edges? <10, <50, <200, >>>>>>>> <1000? >>>>>>>> >>>>>>>> >>>>>>>>> 2. Yes, I tried plocal minutes after posting (d'oh!). I saw a good >>>>>>>>> improvement. It started about 3 times faster and got faster still >>>>>>>>> (about 10 >>>>>>>>> times faster) by the time I checked this morning on a job running >>>>>>>>> overnight. However, even though it is now running at about 7k >>>>>>>>> transactions >>>>>>>>> per second, a billion edges is still going to take about 40 hours. >>>>>>>>> So, I >>>>>>>>> ask myself: is there anyway I can make it faster still? >>>>>>>>> >>>>>>>> >>>>>>>> Here it's missing the usage of AUTO-SHARDING INDEX. Example: >>>>>>>> >>>>>>>> accountClass.createIndex("Account.number", >>>>>>>> OClass.INDEX_TYPE.UNIQUE.toString(), (OProgressListener) null, >>>>>>>> (ODocument) null, >>>>>>>> "AUTOSHARDING", new String[] { "number" }); >>>>>>>> >>>>>>>> In this way you should go more in parallel, because the index is >>>>>>>> distributed across all the shards (clusters) of Account class. you >>>>>>>> should >>>>>>>> have 32 of them by default because you have 32 cores. >>>>>>>> >>>>>>>> Please let me know if by sorting the from_accounts and with this >>>>>>>> change if it's much faster. >>>>>>>> >>>>>>>> This is the best you can have out of the box. To push numbers up >>>>>>>> it's slightly more complicated: you should be sure that transactions >>>>>>>> go in >>>>>>>> parallel and they aren't serialized. This is possible by playing with >>>>>>>> internal OrientDB settings (mainly the distributed workerThreads), by >>>>>>>> having many clusters per class (You could try with 128 first and see >>>>>>>> how >>>>>>>> it's going). >>>>>>>> >>>>>>>> >>>>>>>>> I assume when I start the servers up in distributed mode once >>>>>>>>> more, the data will then be distributed across all nodes in the >>>>>>>>> cluster? >>>>>>>>> >>>>>>>> >>>>>>>> That's right. >>>>>>>> >>>>>>>> >>>>>>>>> 3. I'll return to concurrent, remote inserts when this job has >>>>>>>>> finished. Hopefully, a smaller batch size will mean there is no >>>>>>>>> degradation >>>>>>>>> in performance either... FYI: with a somewhat unscientific approach, >>>>>>>>> I was >>>>>>>>> polling the server JVM with JStack and saw only a single thread doing >>>>>>>>> all >>>>>>>>> the work and it *seemed* to spend a lot of its time in ODirtyManager >>>>>>>>> on >>>>>>>>> collection manipulation. >>>>>>>>> >>>>>>>> >>>>>>>> I think it's because you didn't use the AUTO-SHARDING index. >>>>>>>> Furthermore running distributed, unfortunately, means the tree ridbag >>>>>>>> is >>>>>>>> not available (we will support it in the future), so every change to >>>>>>>> the >>>>>>>> edges takes a lot of CPU to demarshall and marshall the entire edge >>>>>>>> list >>>>>>>> everytime you update a vertex. That's why my recommendation about >>>>>>>> sorting >>>>>>>> the vertices. >>>>>>>> >>>>>>>> >>>>>>>>> I totally appreciate that performance tuning is an empirical >>>>>>>>> science, but do you have any opinions as to which would probably be >>>>>>>>> faster: >>>>>>>>> single-threaded plocal or multithreaded remote? >>>>>>>>> >>>>>>>> >>>>>>>> With v2.2 yo can go in parallel, by using the tips above. For sure >>>>>>>> the replication has a cost. I'm sure you can go much faster with just >>>>>>>> one >>>>>>>> node and then start the other 2 nodes to have the database replicated >>>>>>>> automatically. At least for the first massive insertion. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Phillip >>>>>>>>> >>>>>>>> >>>>>>>> Luca >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Wednesday, September 14, 2016 at 3:48:56 PM UTC+1, Phillip >>>>>>>>> Henry wrote: >>>>>>>>>> >>>>>>>>>> Hi, guys. >>>>>>>>>> >>>>>>>>>> I'm conducting a proof-of-concept for a large bank (Luca, we had >>>>>>>>>> a 'phone conf on August 5...) and I'm trying to bulk insert a >>>>>>>>>> humongous >>>>>>>>>> amount of data: 1 million vertices and 1 billion edges. >>>>>>>>>> >>>>>>>>>> Firstly, I'm impressed about how easy it was to configure a >>>>>>>>>> cluster. However, the performance of batch inserting is bad (and >>>>>>>>>> seems to >>>>>>>>>> get considerably worse as I add more data). It starts at about 2k >>>>>>>>>> vertices-and-edges per second and deteriorates to about 500/second >>>>>>>>>> after >>>>>>>>>> only about 3 million edges have been added. This also takes ~ 30 >>>>>>>>>> minutes. >>>>>>>>>> Needless to say that 1 billion payments (edges) will take over a >>>>>>>>>> week at >>>>>>>>>> this rate. >>>>>>>>>> >>>>>>>>>> This is a show-stopper for us. >>>>>>>>>> >>>>>>>>>> My data model is simply payments between accounts and I store it >>>>>>>>>> in one large file. It's just 3 fields and looks like: >>>>>>>>>> >>>>>>>>>> FROM_ACCOUNT TO_ACCOUNT AMOUNT >>>>>>>>>> >>>>>>>>>> In the test data I generated, I had 1 million accounts and 1 >>>>>>>>>> billion payments randomly distributed between pairs of accounts. >>>>>>>>>> >>>>>>>>>> I have 2 classes in OrientDB: ACCOUNTS (extending V) and PAYMENT >>>>>>>>>> (extending E). There is a UNIQUE_HASH_INDEX on ACCOUNTS for the >>>>>>>>>> account >>>>>>>>>> number (a string). >>>>>>>>>> >>>>>>>>>> We're using OrientDB 2.2.7. >>>>>>>>>> >>>>>>>>>> My batch size is 5k and I am using the "remote" protocol to >>>>>>>>>> connect to our cluster. >>>>>>>>>> >>>>>>>>>> I'm using JDK 8 and my 3 boxes are beefy machines (32 cores each) >>>>>>>>>> but without SSDs. I wrote the importing code myself but did nothing >>>>>>>>>> 'clever' (I think) and used the Graph API. This client code has been >>>>>>>>>> given >>>>>>>>>> lots of memory and using jstat I can see it is not excessively GCing. >>>>>>>>>> >>>>>>>>>> So, my questions are: >>>>>>>>>> >>>>>>>>>> 1. what kind of performance can I realistically expect and can I >>>>>>>>>> improve what I have at the moment? >>>>>>>>>> >>>>>>>>>> 2. what kind of degradation should I expect as the graph grows? >>>>>>>>>> >>>>>>>>>> Thanks, guys. >>>>>>>>>> >>>>>>>>>> Phillip >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "OrientDB" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to orient-databa...@googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "OrientDB" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to orient-databa...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OrientDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to orient-databa...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to orient-databa...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> > Best regards, >> Andrey Lomakin, R&D lead. >> OrientDB Ltd >> >> twitter: @Andrey_Lomakin >> linkedin: https://ua.linkedin.com/in/andreylomakin >> blogger: http://andreylomakin.blogspot.com/ >> > -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to orient-database+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- Best regards, Andrey Lomakin, R&D lead. OrientDB Ltd twitter: @Andrey_Lomakin linkedin: https://ua.linkedin.com/in/andreylomakin blogger: http://andreylomakin.blogspot.com/ -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
