LOL, there's lots of things where I'd like to "move the problem elsewhere".
I've achieved concurrent 120K on the server hardware but it depends on the input. There's another recent Jena thread regarding sizing and that's tied up with what's in the input. I see the same thing with loading data, some files fly others seem to drag and it's not just the size. What the server hardware does do is allow me to run multiple processes and average 60K. Also up to a certain size I have an overclocked AMD (4.5Ghz) and it will outperform everything until it hits its cache limit. We tend towards running multiple TDB's and present them as one, a legacy of overcoming the one writer in TDB1. This brings it's own issues such as distinct being high cost which we mitigate with a few tricks. On the minefield subject of hardware, do you have DDR3 or DDR4? What chipset is driving it because Haswell’s dual-channel memory controller is going to have a hard time keeping up with the quad-channel memory controllers on Ivy Bridge-E and Haswell-E. And yes Corsair quote 47GB/s for DDR4, but you still need to write that somewhere and a M.2 a PCI-E 2.0 x4 at 1.6GB/s is almost 3x the througput of SATAIII at 600MB/s, PCI-E 3.0 x4 is 3.9GB/s, plus you now have Optane or 3D XPoint depending on what sounds better What files are you trying to import and i'll run them through? Regards Dick On 28 November 2017 at 15:30, Laura Morales <[email protected]> wrote: > > Eventually something will give and you'll get a wait as something is > spilled to something, ie cache to physical drive. > > Also different settings suit different work loads. I have a number of > +128GB units configured differently depending on what they need to do. The > ETL setting only gives Java 8GB but the OS will consume close to 90GB > virtual for the process as it basically dumps into file cache. At some > point though that cache is written out to noon volatile storage. As the > units have 24 cores I can actually run close to 12 processes before things > start to effect each other. If you consider server class hardware there's a > lot of thought to cache levels and how they cascade. > > Switch the SATA for M.2 and you'll move the issue somewhere else... > > Well yeah, but having a problem at 10K triples/seconds is not the same > problem as 1M triples/seconds. I'll gladly "move the problem elsewhere" if > I knew how to get to 1M triples/seconds. > Moving from SATA to M.2 I don't know if it's worth the trouble (and money) > given that on my computer running from SATA3 disks or RAMdisk doesn't seem > like it's making any difference. And RAM is much faster than M.2 too. > Just out of curiosity, how many "AVG triples/seconds" can you get with > your server-class hardware when converting a .nt to TDB2 using > tdb2.tdbloader? >
