[
https://issues.apache.org/jira/browse/JENA-1552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16499982#comment-16499982
]
ASF GitHub Bot commented on JENA-1552:
--------------------------------------
Github user rvesse commented on a diff in the pull request:
https://github.com/apache/jena/pull/426#discussion_r192680143
--- Diff:
jena-db/jena-tdb2/src/main/java/org/apache/jena/tdb2/loader/parallel/LoaderPlan.java
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.jena.tdb2.loader.parallel;
+
+import org.apache.jena.tdb2.store.NodeId;
+
+/**
+ * A {@code LoaderPlan}
+ * <p>
+ * For triples and for quads there is a first phase to parse the input,
+ * convert to tuples of {@link NodeId NodeIds}, including allocating the
ids,
+ * and do at least one tuple index for each of triples quads to capture
the input.
+ * <p>
+ * After that, a number of phases builds the other indexes.
+ * <p>
+ * The {@code mulithreadedInput} f;ag indicates whether the first phase is
--- End diff --
Typo `f;ag` -> `flag`
> Bulk loader for TDB2 (phased loading)
> -------------------------------------
>
> Key: JENA-1552
> URL: https://issues.apache.org/jira/browse/JENA-1552
> Project: Apache Jena
> Issue Type: Improvement
> Components: TDB2
> Reporter: Andy Seaborne
> Assignee: Andy Seaborne
> Priority: Major
>
> Following on from JENA-1550, this ticket is for phased loading which combined
> features of the sequential loader and the parallel loader.
> When building all the persistent datastructures (parallel loader), the work
> on different indexes at the same time is competing for hardware resources,
> RAM and I/O bandwidth. As the size to load grows, this becomes a noticeable
> slowdown.
> The sequential loader is the other extreme of the design spectrum. It does
> work on one index at a time so as to maximize caching efficiency.
> Phased loading has parallel operation per phase and splits work into subsets
> of indexes.
> At 200m and loading to rotational disk, an experimental phased loader working
> with 2 indexes at a time, starts to become faster than parallel on the same
> hardware as used for the [figures in
> JENA-1550|https://issues.apache.org/jira/browse/JENA-1550#comment-16484269]
> (57K parallel, 70K phased).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
