>From Ritik Raj <[email protected]>: Ritik Raj has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20588?usp=email )
Change subject: [ASTERIXDB-3652][STO] Restore the old schema on abort ...................................................................... [ASTERIXDB-3652][STO] Restore the old schema on abort - user model changes: no - storage format changes: no - interface changes: no Details: After abort, restore the previous schema for the index. Ext-ref: MB-69414 Change-Id: I97ba7dbc39241ffba2554ea92ee49be608b28cde Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20588 Tested-by: Jenkins <[email protected]> Integration-Tests: Jenkins <[email protected]> Reviewed-by: Murtadha Hubail <[email protected]> --- A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.update.sqlpp R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.005.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.update.sqlpp R asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.005.query.sqlpp R asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.005.adm R asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.005.adm M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnImmutableMetadata.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnMetadata.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/schema/ObjectSchemaNode.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/ColumnTransformer.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/FlushColumnMetadata.java M asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/NoWriteColumnTransformer.java M hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/util/ArrayBackedValueStorage.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnMetadata.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/lsm/LSMColumnIndexBulkloader.java 15 files changed, 100 insertions(+), 18 deletions(-) Approvals: Jenkins: Verified; Verified Anon. E. Moose #1000171: Murtadha Hubail: Looks good to me, approved diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.update.sqlpp new file mode 100644 index 0000000..075d204 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.update.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +USE test; + +COMPACT DATASET ColumnDataset; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.005.query.sqlpp similarity index 100% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.query.sqlpp rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.005.query.sqlpp diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.update.sqlpp new file mode 100644 index 0000000..075d204 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.update.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +USE test; + +COMPACT DATASET ColumnDataset; \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.005.query.sqlpp similarity index 100% rename from asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.query.sqlpp rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.005.query.sqlpp diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.005.adm similarity index 100% rename from asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.004.adm rename to asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-4/ASTERIXDB-3652.005.adm diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.005.adm similarity index 100% rename from asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.004.adm rename to asterixdb/asterix-app/src/test/resources/runtimets/results/column/assembly/ASTERIXDB-3652-5/ASTERIXDB-3652.005.adm diff --git a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnImmutableMetadata.java b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnImmutableMetadata.java index c7b4651..9754c75 100644 --- a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnImmutableMetadata.java +++ b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnImmutableMetadata.java @@ -39,6 +39,11 @@ } @Override + public final void swapSerializedColumnsMetadata() { + // since the metadata is immutable + } + + @Override public final void abort() throws HyracksDataException { //NoOp as the metadata is immutable } diff --git a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnMetadata.java b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnMetadata.java index 2e95384..3637a29 100644 --- a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnMetadata.java +++ b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/AbstractColumnMetadata.java @@ -63,4 +63,4 @@ @Override public abstract int getNumberOfColumns(); -} \ No newline at end of file +} diff --git a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/schema/ObjectSchemaNode.java b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/schema/ObjectSchemaNode.java index 59a3edd..6b90ade 100644 --- a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/schema/ObjectSchemaNode.java +++ b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/metadata/schema/ObjectSchemaNode.java @@ -49,7 +49,6 @@ private final Int2IntMap fieldNameIndexToChildIndexMap; private final List<AbstractSchemaNode> children; private IntUnaryOperator nextIndex; - private boolean isEmptyObject = false; public ObjectSchemaNode() { fieldNameIndexToChildIndexMap = new Int2IntOpenHashMap(); @@ -81,7 +80,7 @@ FlushColumnMetadata columnMetadata) throws HyracksDataException { int numberOfChildren = children.size(); int fieldNameIndex = columnMetadata.getFieldNamesDictionary().getOrCreateFieldNameIndex(fieldName); - boolean previouslyMissing = isEmptyObject; + boolean previouslyMissing = isEmptyMissingObject(); int childIndex = fieldNameIndexToChildIndexMap.getOrDefault(fieldNameIndex, nextIndex.apply(fieldNameIndex)); AbstractSchemaNode currentChild = childIndex == numberOfChildren ? null : children.get(childIndex); AbstractSchemaNode newChild = columnMetadata.getOrCreateChild(currentChild, childTypeTag, previouslyMissing); @@ -111,13 +110,17 @@ if (!children.isEmpty()) { return null; } - isEmptyObject = true; + AbstractSchemaNode emptyChild = columnMetadata.getOrCreateChild(null, ATypeTag.MISSING, false); addChild(DUMMY_FIELD_NAME_INDEX, emptyChild); nextIndex = this::emptyColumnIndex; return emptyChild; } + public boolean isEmptyMissingObject() { + return children.size() == 1 && fieldNameIndexToChildIndexMap.containsKey(DUMMY_FIELD_NAME_INDEX); + } + public AbstractSchemaNode getChild(int fieldNameIndex) { if (fieldNameIndexToChildIndexMap.containsKey(fieldNameIndex)) { return children.get(fieldNameIndexToChildIndexMap.get(fieldNameIndex)); @@ -217,11 +220,6 @@ nextIndex = this::nextIndex; fieldNameIndexToChildIndexMap.remove(DUMMY_FIELD_NAME_INDEX); fieldNameIndexToChildIndexMap.put(fieldNameIndex, 0); - isEmptyObject = false; return 0; } - - public boolean isEmptyObject() { - return isEmptyObject; - } } diff --git a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/ColumnTransformer.java b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/ColumnTransformer.java index 7b1a85b..d146872 100644 --- a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/ColumnTransformer.java +++ b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/ColumnTransformer.java @@ -130,7 +130,7 @@ if (pointable.getNumberOfChildren() == 0) { // Set as empty object objectNode.setEmptyObject(columnMetadata); - if (!objectNode.isMissingInitiallyInBatch() && objectNode.isEmptyObject()) { + if (!objectNode.isMissingInitiallyInBatch() && objectNode.isEmptyMissingObject()) { objectNode.needAllColumns(true); objectNode.setMissingInitiallyInBatch(true); PrimitiveSchemaNode missingNode = (PrimitiveSchemaNode) objectNode.getChildren().get(0); diff --git a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/FlushColumnMetadata.java b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/FlushColumnMetadata.java index 2303235..17a0607 100644 --- a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/FlushColumnMetadata.java +++ b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/FlushColumnMetadata.java @@ -82,10 +82,12 @@ private final IColumnValuesWriterFactory columnWriterFactory; protected final List<IColumnValuesWriter> columnWriters; private final ArrayBackedValueStorage serializedMetadata; + private final ArrayBackedValueStorage tempSerializedMetadata; private final PathInfoSerializer pathInfoSerializer; protected final IntArrayList nullWriterIndexes; private final boolean metaContainsKeys; private boolean changed; + private boolean schemaEvolved; protected int level; protected int repeated; protected int requiredTemporaryBuffersCount; @@ -125,8 +127,11 @@ } serializedMetadata = new ArrayBackedValueStorage(); + tempSerializedMetadata = new ArrayBackedValueStorage(); + schemaEvolved = false; changed = true; serializeColumnsMetadata(); + swapSerializedColumnsMetadata(); } public FlushColumnMetadata(ARecordType datasetType, ARecordType metaType, int numPrimaryKeys, @@ -153,6 +158,8 @@ //Add definition levels for the root addDefinitionLevelsAndGet(root); this.serializedMetadata = serializedMetadata; + this.tempSerializedMetadata = new ArrayBackedValueStorage(); + this.schemaEvolved = false; changed = false; } @@ -178,17 +185,20 @@ try { serializeChanges(); logSchema(root, metaRoot, fieldNamesDictionary); + schemaEvolved = true; changed = false; } catch (IOException e) { throw HyracksDataException.create(e); } + } else if (!schemaEvolved) { + return serializedMetadata; } - return serializedMetadata; + return tempSerializedMetadata; } private void serializeChanges() throws IOException { - serializedMetadata.reset(); - DataOutput output = serializedMetadata.getDataOutput(); + tempSerializedMetadata.reset(); + DataOutput output = tempSerializedMetadata.getDataOutput(); int writersOffsetPointer = reserveInt(output); int fieldNamesOffsetPointer = reserveInt(output); @@ -222,15 +232,24 @@ pathInfoSerializer.serialize(output, getNumberOfColumns()); } + @Override + public void swapSerializedColumnsMetadata() { + if (schemaEvolved) { + ArrayBackedValueStorage.swap(serializedMetadata, tempSerializedMetadata); + tempSerializedMetadata.reset(); + schemaEvolved = false; + } + } + private int reserveInt(DataOutput output) throws IOException { - int offset = serializedMetadata.getLength(); + int offset = tempSerializedMetadata.getLength(); output.writeInt(-1); return offset; } private void setOffset(int pointer) { - int offset = serializedMetadata.getLength(); - IntegerPointable.setInteger(serializedMetadata.getByteArray(), pointer, offset); + int offset = tempSerializedMetadata.getLength(); + IntegerPointable.setInteger(tempSerializedMetadata.getByteArray(), pointer, offset); } public static FlushColumnMetadata create(ARecordType datasetType, ARecordType metaType, int numPrimaryKeys, @@ -283,6 +302,8 @@ serializedMetadata.getStartOffset(), serializedMetadata.getLength())); try { abort(input); + schemaEvolved = false; + changed = false; } catch (IOException e) { throw HyracksDataException.create(e); } diff --git a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/NoWriteColumnTransformer.java b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/NoWriteColumnTransformer.java index 89d812b..f67ce1c 100644 --- a/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/NoWriteColumnTransformer.java +++ b/asterixdb/asterix-column/src/main/java/org/apache/asterix/column/operation/lsm/flush/NoWriteColumnTransformer.java @@ -110,7 +110,7 @@ if (pointable.getNumberOfChildren() == 0) { // Set as empty object AbstractSchemaNode missingChild = objectNode.setEmptyObject(columnMetadata); - if (!objectNode.isMissingInitiallyInBatch() && objectNode.isEmptyObject()) { + if (!objectNode.isMissingInitiallyInBatch() && objectNode.isEmptyMissingObject()) { objectNode.needAllColumns(true); // to include the missing column, while finalizing the batch. objectNode.setMissingInitiallyInBatch(true); if (missingChild != null) { diff --git a/hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/util/ArrayBackedValueStorage.java b/hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/util/ArrayBackedValueStorage.java index d4feff6..02af029 100644 --- a/hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/util/ArrayBackedValueStorage.java +++ b/hyracks-fullstack/hyracks/hyracks-data/hyracks-data-std/src/main/java/org/apache/hyracks/data/std/util/ArrayBackedValueStorage.java @@ -29,7 +29,7 @@ public class ArrayBackedValueStorage implements IMutableValueStorage, IPointable { - private final GrowableArray data; + private GrowableArray data; public ArrayBackedValueStorage(int size) { data = new GrowableArray(size); @@ -39,6 +39,13 @@ data = new GrowableArray(); } + public static void swap(ArrayBackedValueStorage serializedMetadata, + ArrayBackedValueStorage tempSerializedMetadata) { + GrowableArray temp = serializedMetadata.data; + serializedMetadata.data = tempSerializedMetadata.data; + tempSerializedMetadata.data = temp; + } + @Override public void reset() { data.reset(); diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnMetadata.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnMetadata.java index f11ef9a..0e970b6 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnMetadata.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/api/IColumnMetadata.java @@ -42,4 +42,10 @@ * abort in case of an error. This should clean up any artifact */ void abort() throws HyracksDataException; + + /** + * make the temporary schema the persistent schema + * This happens after the flush is successful. + */ + void swapSerializedColumnsMetadata(); } diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/lsm/LSMColumnIndexBulkloader.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/lsm/LSMColumnIndexBulkloader.java index ba41227..ebd9b63 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/lsm/LSMColumnIndexBulkloader.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree-column/src/main/java/org/apache/hyracks/storage/am/lsm/btree/column/impls/lsm/LSMColumnIndexBulkloader.java @@ -40,6 +40,7 @@ public void end() throws HyracksDataException { ColumnUtil.putColumnsMetadataValue(columnMetadata.serializeColumnsMetadata(), componentMetadata); super.end(); + columnMetadata.swapSerializedColumnsMetadata(); } @Override -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20588?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: asterixdb Gerrit-Branch: phoenix Gerrit-Change-Id: I97ba7dbc39241ffba2554ea92ee49be608b28cde Gerrit-Change-Number: 20588 Gerrit-PatchSet: 6 Gerrit-Owner: Ritik Raj <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Murtadha Hubail <[email protected]> Gerrit-Reviewer: Ritik Raj <[email protected]>
