Update: The delete ran for more than 6 hours before I manually killed the connection via post to http://192.168.1.13:2480/connection/kill/87. Upon further inspection, this vertex had more than 75,000 edges (very unusual for our app) which I suspect is why the server ran out of memory.
Sorry the server log was automatically deleted and if the heap dump file is /usr/bin/orientdb-community-2.1.0/bin/java_pid1357.hprof then it's just under 2GB. My takeaways from this experience are a) be very careful deleting things. If you delete an RID manually, make sure to interrogate the item (number of edges, etc) to determine potential impact before you issue the command. If you allow end users to delete RIDs via web interface or have scheduled processes that do it then you are completely hosed. b) when delete query fails due to out of memory, it seems orientdb tries the delete over and over again never-ending (per the server log) which exacerbates the problem. If delete runs out of memory on the first attempt why would it succeed on the 500th attempt? c) deleting one RID can definitely bring your whole database down for hours, affecting all end users by making their binary connections unusable. No known workaround. d) if you can isolate the problem and figure out which exact connection ID is the culprit then doing a /connection/kill/<id> seems to resolve the situation without having to resort to a kill -9 at the operating system level and restoring the database from backup. On Thu, Aug 27, 2015 at 1:03 AM, Andrey Lomakin <[email protected]> wrote: > Hi, > Could you send: > > 1. Server log. > 2. Heap dump, it is created in server installation directory once OOM is > thrown. > > On Thu, Aug 27, 2015 at 2:42 AM David Carr <[email protected]> wrote: > >> Been down for 90 minutes now while this delete continues to run. I never >> know whether to wait it out or kill -9 the server process and deal with >> data corruption (usually have to restore from backup). Will wait it out a >> bit longer and see what happens. >> >> >> On Wednesday, August 26, 2015 at 4:32:36 PM UTC-6, David Carr wrote: >>> >>> Doing data cleanup on orientdb 2.1.0... >>> >>> I issued console command "delete vertex #21:121048" and for the next 60 >>> minutes any other (binary in our case) client connections receive this >>> message: >>> >>> *ERROR socket_read(): unable to read from socket [10060]: A connection >>> attempt failed because the connected party did not properly respond after a >>> period of time, or established connection failed because connected host has >>> failed to respond.* >>> >>> >>> When the delete completed all clients could connect normally again. >>> Orientdb server log files contain (repeatedly): >>> >>> *java.net.SocketException: Broken pipe* >>> >>> *Error during WAL background flush * >>> >>> *java.lang.OutOfMemoryError: Java heap space* >>> >>> >>> This is a real problem to take down an entire db host just to do data >>> cleanup on one vertex. I'm not sure how many edges it had but presumably >>> there must have been a lot to cause this big of a problem. Will make >>> scheduling periodic maintenance a real burden for IT and hardship for end >>> users. >>> >>> Anybody else out there seeing this type of system behavior? >>> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "OrientDB" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/orient-database/7TjsyjMqpsk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- David Carr 801.231.4946 -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
