Neunhoef added a comment. > OK. Maybe its just a function of not using a super nice machine for testing. > We really do want the system to scale down to work with less ram and cheap, > big spinning disks so folks can run it on their laptop. That'll encourage > experimentation.
Well, ArangoDB is designed to be a "mostly-in-memory" database, which means that it really likes to keep all its data including indexes in main memory. The actual data is persisted using memory mapped files, but all access patterns essentially assume that all data is available in RAM. By the way, most of your candidate graph databases (Neo4j, OrientDB, Wikidata Query service) use this approach. Only Titan/Cassandra would run well with less RAM than data. Therefore, if you want this, you have to use a corresponding engine, and this will probably mean Titan. However, you also said that you want to index by essentially all attributes. If this is true, then you will face a dilemma with any database solution: Either at least some queries are untolerably slow, because the actual data or the particular index you access needs to be swapped in, or everything fits into RAM, in which case it will be cached there. This is because you have essentially unpredictable random accesses to your data. So I would expect that even with an essentially disk-based database engine you will run into problems and due to the virtualisation features of modern OSes you will get the same performance behaviour with a mostly-in-memory database like ArangoDB for your scenario. TASK DETAIL https://phabricator.wikimedia.org/T88549 REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>. EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Smalyshev, Neunhoef Cc: Neunhoef, Fceller, JanZerebecki, Aklapper, Manybubbles, jkroll, Smalyshev, Wikidata-bugs, aude, GWicke, daniel _______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
