>From Ali Alsuliman <[email protected]>: Ali Alsuliman has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21668?usp=email )
Change subject: [ASTERIXDB-3817][MTD] Give VTree the dataset's atomicity ...................................................................... [ASTERIXDB-3817][MTD] Give VTree the dataset's atomicity VTreeResourceFactoryProvider passed a literal false for the LSMVTreeLocalResourceFactory atomic argument, where BTree and ArrayBTree both pass dataset.isAtomic(). The same call site already took dataset.getIoOperationCallbackFactory(index), which returns AtomicLSMIndexIOOperationCallbackFactory on an atomic dataset -- so the index was wired atomic everywhere except its own isAtomic(). On a schemaless collection that inconsistency is fatal, not cosmetic. The modification callback is NoOpOperationCallbackFactory, so incrementNumActiveOperations skips it and numActiveOperations stays 0 for the whole COPY INTO; canSafelyFlush() is therefore always true and the global VBC's flush thread walks in mid-ingest by design. flushIfRequested then iterates the partition's open indexes, and on one whose memory component still holds writers it returns quietly if the index is atomic and throws IllegalStateException otherwise. That throw unwinds out of GlobalVirtualBufferCache$FlushThread.scheduleFlush into a catch (Throwable) whose handler is ExitUtil.halt(EC_IO_SCHEDULER_FAILED) -- a JVM halt, taking the NC down mid-ingest. Reaching it takes VBC usage at flushPageBudget and the VTree visited before the primary, which getDatasetPartitionOpenIndexes leaves to HashSet order, so it presents as an intermittent crash under load rather than a deterministic one. Ext-ref: MB-73717 Co-Authored-By: Claude Opus 5 <[email protected]> Change-Id: Iede3635fac804abfabb9fd6512f12f231ceb0ef8 --- M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java 1 file changed, 9 insertions(+), 1 deletion(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/68/21668/1 diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java index d7ceba3..3b58e62 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java +++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/VTreeResourceFactoryProvider.java @@ -62,6 +62,7 @@ import org.apache.hyracks.storage.am.vector.utils.CrossPollinationConfig; import org.apache.hyracks.storage.common.IResourceFactory; import org.apache.hyracks.storage.common.IStorageManager; +import org.apache.hyracks.util.annotations.AiProvenance; public class VTreeResourceFactoryProvider implements IResourceFactoryProvider { @@ -71,6 +72,7 @@ } @Override + @AiProvenance(agent = AiProvenance.Agent.CLAUDE_OPUS_5, tool = AiProvenance.Tool.CLAUDE_CODE_UI, contributionKind = AiProvenance.ContributionKind.ASSISTED, notes = "Propagate the dataset's atomicity to the VTree resource") public IResourceFactory getResourceFactory(MetadataProvider mdProvider, Dataset dataset, Index index, ARecordType recordType, ARecordType metaType, ILSMMergePolicyFactory mergePolicyFactory, Map<String, String> mergePolicyProperties, ITypeTraits[] filterTypeTraits, @@ -162,11 +164,17 @@ // Distance-function factory — persisted on the resource so a restarted index reconstructs // the same distance implementation. VectorDistanceFunctionFactory distanceFunctionFactory = new VectorDistanceFunctionFactory(distanceMetric); + // Atomicity is a property of the dataset, so every index on it must report the same value -- + // PrimaryIndexOperationTracker#flushIfRequested walks the partition's open indexes, and on one + // whose memory component still holds writers it returns quietly only if that index is atomic, + // and throws otherwise. Under an atomic statement those writers are the normal state, and the + // throw unwinds into GlobalVirtualBufferCache$FlushThread, which halts the JVM. + boolean atomic = dataset.isAtomic(); return new LSMVTreeLocalResourceFactory(storageManager, typeTraits, cmpFactories, filterTypeTraits, filterCmpFactories, filterFields, opTrackerFactory, ioOpCallbackFactory, pageWriteCallbackFactory, metadataPageManagerFactory, vbcProvider, ioSchedulerProvider, mergePolicyFactory, mergePolicyProperties, true, vectorDimensions, vectorFields, - typeTraitProvider.getTypeTrait(BuiltinType.ANULL), NullIntrospector.INSTANCE, false, + typeTraitProvider.getTypeTrait(BuiltinType.ANULL), NullIntrospector.INSTANCE, atomic, vectorAccessorFactory, numPrimaryKeys, numIncludeFields, dataTupleBuilderFactory, distanceFunctionFactory, crossPollination); } else { -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21668?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Iede3635fac804abfabb9fd6512f12f231ceb0ef8 Gerrit-Change-Number: 21668 Gerrit-PatchSet: 1 Gerrit-Owner: Ali Alsuliman <[email protected]>
