>From Ali Alsuliman <[email protected]>:
Ali Alsuliman has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21685?usp=email )
Change subject: [ASTERIXDB-3817][STO] Validate the bulk-load grouping contract
......................................................................
[ASTERIXDB-3817][STO] Validate the bulk-load grouping contract
VTreeBulkLoader.add requires its input grouped by centroid id. The
producer supplies it by sorting on sortFields = {1, 0} with an ascending
integer comparator on field 1, and that was the whole of the
enforcement: an assert next to the sort, and "centroidId MUST be at
index 1" comments in three files. Nothing connected the producer's field
order to the consumer's expectation, and the loader never checked what
arrived.
A violation did not fail. Returning to an already-loaded cluster made
the loader open a second directory chain and record its head over the
first, so the first chain's pages were written and then referenced by
nothing. end()'s second pass points every leaf tuple of the cluster at
the surviving chain, so the records in the orphaned chain became
unreachable for the life of the component -- no exception, nothing in
the log, and a query against that cluster quietly returning a subset.
Nothing above this level notices: the component is structurally valid,
and a recall drop is not something an assertion about returned records
can see.
Three checks, at two levels:
add() rejects a centroid id below the current one, naming both ids. This
is the contract check, placed where the offending tuple is still in
hand. Non-decreasing rather than increasing -- repeats within a group
are the normal case.
clusterIndexOf() bounds the id against the static structure's leaf
range. loadToNextLeafCluster already checked the index it was handed,
but the first tuple of a load never goes through it -- it assigns
currentLeafClusterIndex directly -- so an out-of-range first id surfaced
at flush time as
ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 4
out of clusterFirstDirPageId[], a raw JDK exception naming an array,
arbitrarily far from the tuple responsible.
finalizeClusterDirectory() refuses to record a second directory head for
a cluster that already has one. Not redundant with add(): the check
above covers input, and loadToNextLeafCluster is public, so a caller
driving the loader directly can jump backwards without passing through
add() at all.
The distance ordering within a group is deliberately not checked. It is
not a correctness invariant -- add() places each tuple through
insertSorted, which finds the slot by distance regardless of arrival
order (asserted directly by VTreeDataFrameTest) -- so out-of-order
distances cost page compactions, not a mis-ordered page. The javadoc
says so, to keep the wrong check from being added later.
VTreeBulkLoaderGroupingTest covers all of it against a static structure
built by the real builder. Four of the six tests fail without the
guards, every one of them because nothing was thrown: two adds that
silently succeeded, an out-of-range id that reached end(), and -- the
case this change exists for -- a backward jump through the public switch
after which end() reported success while orphaning a chain. The two that
pass either way are the happy path, kept as a regression guard, and the
switch path that loadToNextLeafCluster already covered.
What the last test asserts is that the guard fires, not that the
orphaned records are unreachable; showing that needs a search over the
loaded component, which is LSM-level. The inference comes from end()'s
second pass, and the test establishes only that the unfixed path was
silent.
Also drops two fully-qualified ErrorCode references in this file, which
is already importing the class.
Ext-ref: MB-73194
Co-Authored-By: Claude Opus 5 <[email protected]>
Change-Id: I7aae838fbe75d2b8d8af08d775cb4d18a45a88ac
---
M
hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
A
hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-vtree-test/src/test/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoaderGroupingTest.java
2 files changed, 413 insertions(+), 9 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/85/21685/1
diff --git
a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
index 143caf7..eb4ce4a 100644
---
a/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
+++
b/hyracks-fullstack/hyracks/hyracks-storage-am-vtree/src/main/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoader.java
@@ -199,8 +199,55 @@
return IntegerPointable.getInteger(tuple.getFieldData(cidField),
tuple.getFieldStart(cidField));
}
+ /**
+ * Translate a centroid id into its cluster index, rejecting an id outside
the static structure's leaf
+ * range. {@link #loadToNextLeafCluster} bounds-checks the index it is
handed, but the first tuple of a
+ * load never goes through it -- it assigns {@link
#currentLeafClusterIndex} directly -- so without this
+ * a bad first id surfaces as an ArrayIndexOutOfBoundsException out of
{@code clusterFirstDirPageId[]}
+ * when the cluster is finalized, arbitrarily far from the tuple that
caused it.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.GENERATED)
+ private int clusterIndexOf(int centroidId) throws HyracksDataException {
+ int clusterIndex = centroidId - firstLeafCentroidId;
+ if (clusterIndex < 0 || clusterIndex >= numLeafCentroid) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "Centroid id " + centroidId + " is outside the static
structure's leaf centroid range ["
+ + firstLeafCentroidId + ", " +
(firstLeafCentroidId + numLeafCentroid - 1) + "]");
+ }
+ return clusterIndex;
+ }
+
+ /**
+ * Enforce the grouping half of this loader's input contract: tuples
arrive grouped by centroid id, in
+ * non-decreasing id order. The producer provides it by sorting on {@code
sortFields = {1, 0}} with an
+ * ascending integer comparator on field 1 (see {@code
SecondaryVectorOperationsHelper}); nothing in the
+ * type system ties that field order to this loader's expectation, so it
is checked here.
+ * <p>
+ * Returning to an already-loaded cluster is the case that matters,
because it does not fail on its own.
+ * The loader opens a second directory chain for the cluster and {@link
#finalizeClusterDirectory}
+ * overwrites the recorded head with the new chain's, so the first chain's
pages are written and then
+ * referenced by nothing. {@link #end()}'s second pass then points every
leaf tuple of that cluster at
+ * the surviving chain, and the records in the orphaned chain become
permanently unreachable -- with no
+ * exception, and nothing in the log.
+ * <p>
+ * The distance ordering <em>within</em> a group is deliberately not
checked: it is not a correctness
+ * invariant. {@link #add} places each tuple through {@code insertSorted},
which finds the slot by
+ * distance regardless of arrival order, so out-of-order distances cost
page compactions rather than a
+ * mis-ordered page.
+ */
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.GENERATED)
+ private void requireNonDecreasing(int tupleCentroidId) throws
HyracksDataException {
+ if (tupleCentroidId < currentCentroidId) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "Bulk-load input is not grouped by centroid id: received "
+ tupleCentroidId + " after "
+ + currentCentroidId
+ + ". Input must arrive grouped by centroid id, in
non-decreasing id order.");
+ }
+ }
+
@Override
@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_4_8, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.ASSISTED)
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.ASSISTED, notes = "Validate the centroid-id
grouping contract")
public void add(ITupleReference tuple) throws HyracksDataException {
sampler.addTuple(tuple);
int tupleCentroidId = extractCentroidId(tuple);
@@ -208,15 +255,15 @@
// First tuple being added - initialize for first cluster
LOGGER.log(Level.TRACE, "Starting bulk load with first centroid
cluster: {}", tupleCentroidId);
currentCentroidId = tupleCentroidId;
- currentLeafClusterIndex = tupleCentroidId - firstLeafCentroidId;
+ currentLeafClusterIndex = clusterIndexOf(tupleCentroidId);
createDirectoryPage();
createNewDataPage();
} else if (currentCentroidId != tupleCentroidId) {
+ requireNonDecreasing(tupleCentroidId);
// Moved to a new centroid cluster
LOGGER.log(Level.TRACE, "Switching from centroid {} to centroid
{}", currentCentroidId, tupleCentroidId);
currentCentroidId = tupleCentroidId;
- int targetClusterIndex = tupleCentroidId - firstLeafCentroidId;
- loadToNextLeafCluster(targetClusterIndex);
+ loadToNextLeafCluster(clusterIndexOf(tupleCentroidId));
}
try {
int spaceNeeded = dataFrameTupleWriter.bytesRequired(tuple) +
slotSize;
@@ -227,8 +274,7 @@
int maxUsableTupleBytes = bufferCache.getPageSize() -
currentDataFrame.getPageHeaderSize() - slotSize;
int tupleBytes = dataFrameTupleWriter.bytesRequired(tuple);
if (tupleBytes > maxUsableTupleBytes) {
- throw
HyracksDataException.create(org.apache.hyracks.api.exceptions.ErrorCode.RECORD_IS_TOO_LARGE,
- tupleBytes, maxUsableTupleBytes);
+ throw
HyracksDataException.create(ErrorCode.RECORD_IS_TOO_LARGE, tupleBytes,
maxUsableTupleBytes);
}
if (spaceNeeded > spaceAvailable) {
@@ -259,9 +305,8 @@
*/
public void loadToNextLeafCluster(int targetClusterIndex) throws
HyracksDataException {
if (targetClusterIndex < 0 || targetClusterIndex >= numLeafCentroid) {
- throw
HyracksDataException.create(org.apache.hyracks.api.exceptions.ErrorCode.ILLEGAL_STATE,
- "Target cluster index out of bounds: " +
targetClusterIndex + " (valid range: 0-"
- + (numLeafCentroid - 1) + ")");
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE, "Target
cluster index out of bounds: "
+ + targetClusterIndex + " (valid range: 0-" +
(numLeafCentroid - 1) + ")");
}
if (currentLeafClusterIndex == targetClusterIndex) {
@@ -394,6 +439,7 @@
* the overall write order is: data pages (lower IDs) then directory pages
(higher IDs),
* which naturally maintains strict FIFO ordering.
*/
+ @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.ASSISTED, notes = "Refuse to overwrite a
cluster's recorded directory head")
private void finalizeClusterDirectory() throws HyracksDataException {
// Add current directory page to the pending list
if (currentDirectoryPage != null) {
@@ -431,7 +477,18 @@
write(dirPage);
}
- // Record first directory page ID for this cluster
+ // Record first directory page ID for this cluster. A cluster's chain
is recorded exactly once;
+ // overwriting it would orphan the earlier chain (see
requireNonDecreasing). add() rejects a
+ // returned-to cluster at the source, but loadToNextLeafCluster is
public, so a caller driving the
+ // loader directly can jump backwards without passing through add() --
this catches that path.
+ if (clusterFirstDirPageId[currentLeafClusterIndex] !=
VTreeDataTupleAccessor.UNASSIGNED_DIR_PAGE) {
+ throw HyracksDataException.create(ErrorCode.ILLEGAL_STATE,
+ "Cluster " + currentLeafClusterIndex + " (centroid id "
+ + (firstLeafCentroidId + currentLeafClusterIndex)
+ + ") already has a directory chain starting at
page "
+ + clusterFirstDirPageId[currentLeafClusterIndex]
+ + "; recording a second one would orphan the
first");
+ }
clusterFirstDirPageId[currentLeafClusterIndex] = dirPageIds[0];
pendingDirectoryPages.clear();
diff --git
a/hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-vtree-test/src/test/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoaderGroupingTest.java
b/hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-vtree-test/src/test/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoaderGroupingTest.java
new file mode 100644
index 0000000..e824112
--- /dev/null
+++
b/hyracks-fullstack/hyracks/hyracks-tests/hyracks-storage-am-vtree-test/src/test/java/org/apache/hyracks/storage/am/vector/impls/VTreeBulkLoaderGroupingTest.java
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hyracks.storage.am.vector.impls;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hyracks.api.context.IHyracksTaskContext;
+import org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory;
+import org.apache.hyracks.api.dataflow.value.ISerializerDeserializer;
+import org.apache.hyracks.api.dataflow.value.ITypeTraits;
+import org.apache.hyracks.api.exceptions.HyracksDataException;
+import org.apache.hyracks.api.io.FileReference;
+import org.apache.hyracks.data.std.api.IValueReference;
+import org.apache.hyracks.data.std.primitive.DoublePointable;
+import org.apache.hyracks.data.std.primitive.IntegerPointable;
+import org.apache.hyracks.data.std.primitive.LongPointable;
+import org.apache.hyracks.dataflow.common.comm.io.ArrayTupleBuilder;
+import org.apache.hyracks.dataflow.common.comm.io.ArrayTupleReference;
+import org.apache.hyracks.dataflow.common.data.accessors.ITupleReference;
+import
org.apache.hyracks.dataflow.common.data.marshalling.DoubleArraySerializerDeserializer;
+import
org.apache.hyracks.dataflow.common.data.marshalling.DoubleSerializerDeserializer;
+import
org.apache.hyracks.dataflow.common.data.marshalling.Integer64SerializerDeserializer;
+import
org.apache.hyracks.dataflow.common.data.marshalling.IntegerSerializerDeserializer;
+import org.apache.hyracks.dataflow.common.utils.TupleUtils;
+import org.apache.hyracks.storage.am.common.api.IPageManager;
+import org.apache.hyracks.storage.am.common.api.ITreeIndexFrameFactory;
+import org.apache.hyracks.storage.am.common.api.ITreeIndexMetadataFrame;
+import
org.apache.hyracks.storage.am.common.freepage.LinkedMetadataPageManagerFactory;
+import org.apache.hyracks.storage.am.common.impls.NoOpIndexAccessParameters;
+import
org.apache.hyracks.storage.am.lsm.vector.tuples.LSMVTreeDataTupleWriterFactory;
+import org.apache.hyracks.storage.am.vector.TestDoubleArrayVectorAccessor;
+import org.apache.hyracks.storage.am.vector.TestVTreeDistanceFunctionFactory;
+import org.apache.hyracks.storage.am.vector.frames.VTreeDataFrameFactory;
+import org.apache.hyracks.storage.am.vector.frames.VTreeInteriorFrameFactory;
+import org.apache.hyracks.storage.am.vector.frames.VTreeLeafFrameFactory;
+import org.apache.hyracks.storage.am.vector.frames.VTreeMetadataFrameFactory;
+import org.apache.hyracks.storage.am.vector.utils.CrossPollinationConfig;
+import org.apache.hyracks.storage.am.vector.utils.VTreeMetadataKeys;
+import org.apache.hyracks.storage.common.IIndexBulkLoader;
+import org.apache.hyracks.storage.common.ISketchSampler;
+import org.apache.hyracks.storage.common.buffercache.IBufferCache;
+import org.apache.hyracks.storage.common.buffercache.NoOpPageWriteCallback;
+import org.apache.hyracks.test.support.TestStorageManagerComponentHolder;
+import org.apache.hyracks.test.support.TestUtils;
+import org.apache.hyracks.util.annotations.AiProvenance;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * The input contract of {@link VTreeBulkLoader}: tuples arrive grouped by
centroid id, in non-decreasing
+ * id order. The producer supplies it by sorting on {@code sortFields = {1,
0}} with an ascending integer
+ * comparator on field 1 ({@code SecondaryVectorOperationsHelper}), and
nothing in the type system connects
+ * that field order to this loader's expectation — it was previously carried
by two {@code MUST be at field
+ * index 1} comments and an {@code assert} on the producer side.
+ * <p>
+ * A violation used to be silent, which is why these tests exist. Returning to
an already-loaded cluster
+ * does not fail: the loader opens a second directory chain and records its
head over the first, so the
+ * first chain's pages are written and then referenced by nothing, and {@code
end()}'s second pass points
+ * every leaf tuple of that cluster at the surviving chain. The records in the
orphaned chain are then
+ * unreachable for the life of the component — no exception, nothing in the
log, and a query against that
+ * cluster quietly returns a subset. Nothing above this level notices: the
index is structurally valid and
+ * a recall drop is not something an assertion about returned records can see.
+ * <p>
+ * The out-of-range cases are about diagnosis rather than corruption. An id
outside the static structure's
+ * leaf range did fail, but as an {@code ArrayIndexOutOfBoundsException} out
of {@code
+ * clusterFirstDirPageId[]} at flush time — a raw JDK exception naming an
array, arbitrarily far from the
+ * tuple that caused it.
+ */
+@AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool =
AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind =
AiProvenance.ContributionKind.TEST_GENERATED)
+public class VTreeBulkLoaderGroupingTest {
+
+ private static final int PAGE_SIZE = 512;
+ private static final int NUM_PAGES = 200;
+ private static final int MAX_OPEN_FILES = 10;
+ private static final int FRAME_SIZE = 32768;
+ private static final int DIMENSIONS = 2;
+ private static final int MAX_ENTRIES_PER_PAGE = 8;
+
+ /** Four leaf centroids, one per quadrant — enough clusters to leave and
return to one. */
+ private static final double[][] LEAF_CENTROIDS = { { 10, 10 }, { -10, 10
}, { -10, -10 }, { 10, -10 } };
+
+ private IHyracksTaskContext ctx;
+ private IBufferCache bufferCache;
+ private VTree staticTree;
+ private VTree dataTree;
+ private VTree.VTreeAccessor staticAccessor;
+
+ /** Read out of the static structure's metadata rather than assumed, so
the fixture cannot drift. */
+ private int firstLeafCentroidId;
+ private int numLeafCentroid;
+
+ @Before
+ public void setUp() throws HyracksDataException {
+ ctx = TestUtils.create(FRAME_SIZE);
+ TestStorageManagerComponentHolder.init(PAGE_SIZE, NUM_PAGES,
MAX_OPEN_FILES);
+ bufferCache =
TestStorageManagerComponentHolder.getBufferCache(ctx.getJobletContext().getServiceContext());
+
+ staticTree = newTree(ctx.getIoManager().getFileReference(0,
"vtree-grouping-static"));
+ staticTree.create();
+ staticTree.activate();
+ buildStaticStructure();
+ readCentroidRange();
+
+ dataTree = newTree(ctx.getIoManager().getFileReference(0,
"vtree-grouping-data"));
+ dataTree.create();
+ dataTree.activate();
+ }
+
+ @After
+ public void tearDown() throws HyracksDataException {
+ if (staticAccessor != null) {
+ staticAccessor.destroy();
+ }
+ dataTree.deactivate();
+ dataTree.destroy();
+ staticTree.deactivate();
+ staticTree.destroy();
+ bufferCache.close();
+ }
+
+ /**
+ * The shape the producer's sort actually yields: groups in ascending id
order, with repeats inside a
+ * group. Repeats matter because the invariant is non-decreasing, not
strictly increasing — a check
+ * written as {@code <=} would reject every real load after its first
tuple.
+ */
+ @Test
+ public void groupedInputWithRepeatsWithinAGroupIsAccepted() throws
HyracksDataException {
+ CountingSampler sampler = new CountingSampler();
+ IIndexBulkLoader loader = newLoader(sampler);
+
+ loader.add(dataTuple(0.5, centroid(0), 1L));
+ loader.add(dataTuple(1.5, centroid(0), 2L));
+ loader.add(dataTuple(0.2, centroid(1), 3L));
+ loader.add(dataTuple(0.9, centroid(1), 4L));
+ loader.add(dataTuple(0.1, centroid(2), 5L));
+ loader.end();
+
+ // end() publishes the centroid range onto the loaded component;
reaching it means every cluster
+ // was finalized without the new guard firing on a legitimate load.
+ Assert.assertEquals(numLeafCentroid, readLong(dataTree,
VTreeMetadataKeys.NUM_LEAF_CENTROIDS));
+ Assert.assertEquals(firstLeafCentroidId, readLong(dataTree,
VTreeMetadataKeys.FIRST_LEAF_CENTROID_ID));
+ // Every tuple reaches the sketch. The grouping check sits after that
call, so a future guard moved
+ // ahead of it would silently start sampling a subset of the load.
+ Assert.assertEquals("every added tuple must reach the sketch sampler",
5, sampler.tuples());
+ }
+
+ /**
+ * The case that used to lose records. Leaving cluster 0 for cluster 1 and
then coming back to 0 is
+ * rejected at the tuple that breaks the order, naming both ids.
+ */
+ @Test
+ public void returningToAnEarlierClusterIsRejected() throws
HyracksDataException {
+ IIndexBulkLoader loader = newLoader();
+ loader.add(dataTuple(0.5, centroid(0), 1L));
+ loader.add(dataTuple(0.5, centroid(1), 2L));
+
+ HyracksDataException failure =
+ Assert.assertThrows(HyracksDataException.class, () ->
loader.add(dataTuple(0.7, centroid(0), 3L)));
+
+ Assert.assertTrue("the message should name both ids, got: " +
failure.getMessage(),
+ failure.getMessage().contains(String.valueOf(centroid(0)))
+ &&
failure.getMessage().contains(String.valueOf(centroid(1))));
+ }
+
+ /** Wholly unsorted input is caught on its first backward step, not merely
on a revisit. */
+ @Test
+ public void descendingCentroidIdsAreRejected() throws HyracksDataException
{
+ IIndexBulkLoader loader = newLoader();
+ loader.add(dataTuple(0.5, centroid(2), 1L));
+
+ Assert.assertThrows(HyracksDataException.class, () ->
loader.add(dataTuple(0.5, centroid(1), 2L)));
+ }
+
+ /**
+ * The first tuple of a load never passes through {@code
loadToNextLeafCluster}, so it used to skip that
+ * method's bounds check and set a cluster index that only failed later,
indexing
+ * {@code clusterFirstDirPageId[]}.
+ */
+ @Test
+ public void anOutOfRangeCentroidIdOnTheFirstTupleFailsAtAdd() throws
HyracksDataException {
+ IIndexBulkLoader above = newLoader();
+ HyracksDataException failure =
Assert.assertThrows(HyracksDataException.class,
+ () -> above.add(dataTuple(0.5, firstLeafCentroidId +
numLeafCentroid, 1L)));
+ Assert.assertTrue("the message should name the offending id, got: " +
failure.getMessage(),
+
failure.getMessage().contains(String.valueOf(firstLeafCentroidId +
numLeafCentroid)));
+
+ IIndexBulkLoader below = newLoader();
+ Assert.assertThrows(HyracksDataException.class, () ->
below.add(dataTuple(0.5, firstLeafCentroidId - 1, 2L)));
+ }
+
+ /** The same rejection on the switch path, which had the check but
reported an index, not an id. */
+ @Test
+ public void anOutOfRangeCentroidIdOnASwitchFailsAtAdd() throws
HyracksDataException {
+ IIndexBulkLoader loader = newLoader();
+ loader.add(dataTuple(0.5, centroid(0), 1L));
+
+ Assert.assertThrows(HyracksDataException.class,
+ () -> loader.add(dataTuple(0.5, firstLeafCentroidId +
numLeafCentroid, 2L)));
+ }
+
+ /**
+ * {@code loadToNextLeafCluster} is public, so a caller can drive the
loader backwards without passing
+ * through {@code add} and its ordering check. The guard at the point the
directory head is recorded is
+ * what covers that path — this is the test that distinguishes the two
guards.
+ */
+ @Test
+ public void aBackwardJumpThroughThePublicSwitchIsRejected() throws
HyracksDataException {
+ VTreeBulkLoader loader = newLoader();
+ loader.add(dataTuple(0.5, centroid(0), 1L));
+ loader.add(dataTuple(0.5, centroid(1), 2L));
+
+ // Cluster 0's chain is already recorded; re-opening it must not
overwrite that head.
+ loader.loadToNextLeafCluster(0);
+
+ HyracksDataException failure =
Assert.assertThrows(HyracksDataException.class, loader::end);
+ Assert.assertTrue("the message should say the chain would be orphaned,
got: " + failure.getMessage(),
+ failure.getMessage().contains("orphan"));
+ }
+
+ // ---- fixture
-----------------------------------------------------------------------------
+
+ private int centroid(int clusterIndex) {
+ return firstLeafCentroidId + clusterIndex;
+ }
+
+ private VTreeBulkLoader newLoader() throws HyracksDataException {
+ return newLoader(new CountingSampler());
+ }
+
+ private VTreeBulkLoader newLoader(ISketchSampler sampler) throws
HyracksDataException {
+ if (staticAccessor != null) {
+ staticAccessor.destroy();
+ }
+ staticAccessor = (VTree.VTreeAccessor)
staticTree.createAccessor(NoOpIndexAccessParameters.INSTANCE);
+ return (VTreeBulkLoader)
dataTree.createComponentBulkLoader(NoOpPageWriteCallback.INSTANCE,
staticAccessor,
+ sampler);
+ }
+
+ /** {@code [distance: double, centroidId: int, pk: long]} — the
non-quantized data-tuple layout. */
+ private static ITupleReference dataTuple(double distance, int centroidId,
long primaryKey)
+ throws HyracksDataException {
+ ArrayTupleBuilder tupleBuilder = new ArrayTupleBuilder(3);
+ ArrayTupleReference tupleRef = new ArrayTupleReference();
+ ISerializerDeserializer[] serdes = {
DoubleSerializerDeserializer.INSTANCE,
+ IntegerSerializerDeserializer.INSTANCE,
Integer64SerializerDeserializer.INSTANCE };
+ TupleUtils.createTuple(tupleBuilder, tupleRef, serdes, new Object[] {
distance, centroidId, primaryKey });
+ return tupleRef;
+ }
+
+ private void buildStaticStructure() throws HyracksDataException {
+ List<Integer> clustersPerLevel = Arrays.asList(1);
+ List<List<Integer>> centroidsPerCluster =
Arrays.asList(Arrays.asList(LEAF_CENTROIDS.length));
+
+ IIndexBulkLoader builder =
staticTree.createStaticStructureBulkLoader(1, clustersPerLevel,
centroidsPerCluster,
+ MAX_ENTRIES_PER_PAGE, NoOpPageWriteCallback.INSTANCE);
+ for (int i = 0; i < LEAF_CENTROIDS.length; i++) {
+ builder.add(centroidTuple(i, LEAF_CENTROIDS[i]));
+ }
+ builder.end();
+ // The builder publishes the root through the page manager; copying it
onto the tree is the caller's
+ // step, the way LSMVTreeDiskComponent.setInitialized() does it.
+ staticTree.setRootPageId(staticTree.getPageManager().getRootPageId());
+ }
+
+ private void readCentroidRange() throws HyracksDataException {
+ numLeafCentroid = (int) readLong(staticTree,
VTreeMetadataKeys.NUM_LEAF_CENTROIDS);
+ firstLeafCentroidId = (int) readLong(staticTree,
VTreeMetadataKeys.FIRST_LEAF_CENTROID_ID);
+ Assert.assertEquals("the fixture needs every leaf centroid
addressable", LEAF_CENTROIDS.length,
+ numLeafCentroid);
+ }
+
+ private static long readLong(VTree tree, IValueReference key) throws
HyracksDataException {
+ IPageManager pageManager = tree.getPageManager();
+ ITreeIndexMetadataFrame metaFrame = pageManager.createMetadataFrame();
+ pageManager.getMaxPageId(metaFrame);
+ LongPointable value = LongPointable.FACTORY.createPointable();
+ metaFrame.get(key, value);
+ return value.longValue();
+ }
+
+ /** {@code <cid: int, embedding: double[]>} — the builder assigns the
trailing pointer itself. */
+ private static ITupleReference centroidTuple(int centroidId, double[]
vector) throws HyracksDataException {
+ ArrayTupleBuilder tupleBuilder = new ArrayTupleBuilder(2);
+ ArrayTupleReference tupleRef = new ArrayTupleReference();
+ ISerializerDeserializer[] serdes =
+ { IntegerSerializerDeserializer.INSTANCE,
DoubleArraySerializerDeserializer.INSTANCE };
+ TupleUtils.createTuple(tupleBuilder, tupleRef, serdes, new Object[] {
centroidId, vector });
+ return tupleRef;
+ }
+
+ private VTree newTree(FileReference file) throws HyracksDataException {
+ ITreeIndexFrameFactory interiorFrameFactory = new
VTreeInteriorFrameFactory(DIMENSIONS, null, null);
+ ITreeIndexFrameFactory leafFrameFactory = new
VTreeLeafFrameFactory(DIMENSIONS, false, null, null);
+ ITypeTraits[] dataTraits =
+ { DoublePointable.TYPE_TRAITS, IntegerPointable.TYPE_TRAITS,
LongPointable.TYPE_TRAITS };
+ ITreeIndexFrameFactory dataFrameFactory = new VTreeDataFrameFactory(
+ new LSMVTreeDataTupleWriterFactory(dataTraits, false, null,
null), DIMENSIONS);
+ IPageManager pageManager = new
LinkedMetadataPageManagerFactory().createPageManager(bufferCache);
+ return new VTree(bufferCache, pageManager, interiorFrameFactory,
leafFrameFactory,
+ new VTreeMetadataFrameFactory(DIMENSIONS, null, null),
dataFrameFactory,
+ new IBinaryComparatorFactory[] { null }, DIMENSIONS,
DIMENSIONS, file,
+ TestDoubleArrayVectorAccessor.Factory.INSTANCE, new
VTreeDataTupleBuilderFactory(0, false), null,
+ new TestVTreeDistanceFunctionFactory("euclidean"),
CrossPollinationConfig.LEGACY);
+ }
+
+ /**
+ * The loader only ever calls {@code addTuple}; {@code serialize()} is the
LSM layer's business. It
+ * throws rather than returning null so that a future loader change
calling it shows up here.
+ */
+ private static final class CountingSampler implements ISketchSampler {
+ private int tuples;
+
+ @Override
+ public IValueReference serialize() throws IOException {
+ throw new UnsupportedOperationException("the bulk loader is not
expected to serialize the sketch");
+ }
+
+ @Override
+ public void addTuple(ITupleReference tuple) {
+ tuples++;
+ }
+
+ int tuples() {
+ return tuples;
+ }
+ }
+}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21685?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I7aae838fbe75d2b8d8af08d775cb4d18a45a88ac
Gerrit-Change-Number: 21685
Gerrit-PatchSet: 1
Gerrit-Owner: Ali Alsuliman <[email protected]>
