Andy Seaborne created JENA-1552:
-----------------------------------
Summary: Bulk loader for TDB2 (phased loading)
Key: JENA-1552
URL: https://issues.apache.org/jira/browse/JENA-1552
Project: Apache Jena
Issue Type: Improvement
Components: TDB2
Reporter: Andy Seaborne
Assignee: Andy Seaborne
Following on from JENA-1550, this ticket is for phased loading which combined
features of the sequential loader and the parallel loader.
When building all the persistent datastructures (parallel loader), the work on
different indexes at the same time is competing for hardware resources, RAM and
I/O bandwidth. As the size to load grows, this becomes a noticeable slowdown.
The sequential loader is the other extreme of the design spectrum. It does work
on one index at a time so as to maximize caching efficiency.
Phased loading has parallel operation per phase and splits work into subsets of
indexes.
At 200m and loading to rotational disk, an experimental phased loader working
with 2 indexes at a time, starts to become faster than parallel on the same
hardware as used for the [figures in
JENA-1550|https://issues.apache.org/jira/browse/JENA-1550#comment-16484269]
(56K parallel, 76K phased).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
