Eventually something will give and you'll get a wait as something is spilled to something, ie cache to physical drive. Also different settings suit different work loads. I have a number of +128GB units configured differently depending on what they need to do. The ETL setting only gives Java 8GB but the OS will consume close to 90GB virtual for the process as it basically dumps into file cache. At some point though that cache is written out to noon volatile storage. As the units have 24 cores I can actually run close to 12 processes before things start to effect each other. If you consider server class hardware there's a lot of thought to cache levels and how they cascade. Switch the SATA for M.2 and you'll move the issue somewhere else...
Dick -------- Original message --------From: Laura Morales <[email protected]> Date: 28/11/2017 14:06 (GMT+00:00) To: jena-users-ml <[email protected]> Subject: tdb2.tdbloader performance So I had a laptop at hand with a 3GHz i7 CPU, 8GB DDR3 1600MHz RAM, and SATA3 HDD available. I decided to try the conversion again on a 1.1GB .nt file. I used `./tbd2.tdbloader --loc xxx --verbose file.nt`. Reading the .nt file from the HDD, and writing to the HDD gave me about 60K triples/second on AVG. I don't have an SSD, but this PC seems to have enough RAM. So I started a livecd to be sure that I was running everything from RAM; all disks unmounted. I ran the same command, and the AVG number of triples/seconds is pretty much the same, perhaps only slightly better with 2K or 3K more per seconds. Conversion from the livecd seemed to use a full thread at 100%, 25% RAM, 0 SWAP. This is... very surprising, I wasn't expecting this. I was expecting a significant improvement since I was running everything from RAM. What I get from this is that SATA3 disks are OK? That SSD won't really make any difference? Are faster RAM, faster CPU, or maybe more RAM/CPU cache the only ways to get more performance out of tdb2.tdbloader (since more RAM capacity doesn't seem to make any difference)? Or does tdb2.tdbloader (or maybe Java) have any mechanism in place that is slowing down conversion? Like for example using less RAM than it's available or whatever?
