ibessonov commented on a change in pull request #739:
URL: https://github.com/apache/ignite-3/pull/739#discussion_r833041672
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/index/SortedIndexStorage.java
##########
@@ -58,12 +64,63 @@
*/
void remove(IndexRow row);
+ /** Exclude lower bound. */
+ byte GREATER = 0;
+
+ /** Include lower bound. */
+ byte GREATER_OR_EQUAL = 1;
+
+ /** Exclude upper bound. */
+ byte LESS = 0;
+
+ /** Include upper bound. */
+ byte LESS_OR_EQUAL = 1 << 1;
+
+ byte FORWARD = 0;
+
+ byte BACKWARDS = 1 << 2;
+
/**
* Returns a range of index values between the lower bound (inclusive) and
the upper bound (inclusive).
*/
// TODO: add options https://issues.apache.org/jira/browse/IGNITE-16059
Cursor<IndexRow> range(IndexRowPrefix lowerBound, IndexRowPrefix
upperBound);
+ /**
+ * Returns a range of index values between the lower bound and the upper
bound, consistent with the passed timestamp.
+ *
+ * @param lowerBound Lower bound. Exclusivity is controlled by a {@link
#GREATER_OR_EQUAL} or {@link #GREATER} flag.
+ * {@code null} means unbounded.
+ * @param upperBound Upper bound. Exclusivity is controlled by a {@link
#LESS} or {@link #LESS_OR_EQUAL} flag.
+ * {@code null} means unbounded.
+ * @param flags Control flags. {@link #GREATER} | {@link #LESS} | {@link
#FORWARD} by default. Other available values
+ * are {@link #GREATER_OR_EQUAL}, {@link #LESS_OR_EQUAL} and {@link
#BACKWARDS}.
+ * @param timestamp Timestamp value for consistent multiversioned index
scan.
+ * @param columnsProjection Bit set with column indexes to return. {@code
null} means returning all available columns.
+ * @param partitionFilter Partition filter predicate. {@code null} means
returning data from all partitions.
+ * @return Cursor with fetched index rows.
+ * @throws IllegalArgumentException If backwards flag is passed and
backwards iteration is not supported by the storage.
Review comment:
Good idea, I'll do it
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/index/SortedIndexStorage.java
##########
@@ -58,12 +64,63 @@
*/
void remove(IndexRow row);
+ /** Exclude lower bound. */
+ byte GREATER = 0;
+
+ /** Include lower bound. */
+ byte GREATER_OR_EQUAL = 1;
+
+ /** Exclude upper bound. */
+ byte LESS = 0;
+
+ /** Include upper bound. */
+ byte LESS_OR_EQUAL = 1 << 1;
+
+ byte FORWARD = 0;
+
+ byte BACKWARDS = 1 << 2;
+
/**
* Returns a range of index values between the lower bound (inclusive) and
the upper bound (inclusive).
*/
// TODO: add options https://issues.apache.org/jira/browse/IGNITE-16059
Cursor<IndexRow> range(IndexRowPrefix lowerBound, IndexRowPrefix
upperBound);
+ /**
+ * Returns a range of index values between the lower bound and the upper
bound, consistent with the passed timestamp.
+ *
+ * @param lowerBound Lower bound. Exclusivity is controlled by a {@link
#GREATER_OR_EQUAL} or {@link #GREATER} flag.
+ * {@code null} means unbounded.
+ * @param upperBound Upper bound. Exclusivity is controlled by a {@link
#LESS} or {@link #LESS_OR_EQUAL} flag.
+ * {@code null} means unbounded.
+ * @param flags Control flags. {@link #GREATER} | {@link #LESS} | {@link
#FORWARD} by default. Other available values
+ * are {@link #GREATER_OR_EQUAL}, {@link #LESS_OR_EQUAL} and {@link
#BACKWARDS}.
+ * @param timestamp Timestamp value for consistent multiversioned index
scan.
+ * @param columnsProjection Bit set with column indexes to return. {@code
null} means returning all available columns.
+ * @param partitionFilter Partition filter predicate. {@code null} means
returning data from all partitions.
+ * @return Cursor with fetched index rows.
+ * @throws IllegalArgumentException If backwards flag is passed and
backwards iteration is not supported by the storage.
+ */
+ default Cursor<IndexRowEx> scan(
+ @Nullable IndexRowPrefix lowerBound,
+ @Nullable IndexRowPrefix upperBound,
+ @MagicConstant(flagsFromClass = SortedIndexStorage.class) byte
flags,
+ Timestamp timestamp,
+ @Nullable BitSet columnsProjection,
+ @Nullable IntPredicate partitionFilter
+ ) {
+ throw new UnsupportedOperationException("scan");
+ }
+
+ /**
+ * The sole purpose of this class is to avoid massive refactoring while
changing the original IndexRow.
+ */
+ interface IndexRowEx {
Review comment:
I'm not planning to fix this in current PR, it's too much work. I'll
link a new ticket
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/index/SortedIndexDescriptor.java
##########
@@ -166,7 +166,7 @@ public SortedIndexDescriptor(String name, TableView
tableConfig) {
/**
* Creates a {@link SchemaDescriptor} from a list of index key columns.
*/
- private static SchemaDescriptor createSchemaDescriptor(List<ColumnView>
indexKeyColumnViews) {
+ public static SchemaDescriptor createSchemaDescriptor(List<ColumnView>
indexKeyColumnViews) {
Review comment:
I guess it's an artifact, I made it private again
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestMvPartitionStorage.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Predicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.storage.MvPartitionStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV partition storage.
+ */
+public class TestMvPartitionStorage implements MvPartitionStorage {
+ private static final VersionChain NULL = new VersionChain(null, null,
null, null);
+
+ private final ConcurrentHashMap<ByteBuffer, VersionChain> map = new
ConcurrentHashMap<>();
+
+ private final List<TestSortedIndexMvStorage> indexes;
+
+ public TestMvPartitionStorage(List<TestSortedIndexMvStorage> indexes) {
+ this.indexes = indexes;
+ }
+
+ private static class VersionChain {
+ final BinaryRow row;
+ final Timestamp begin;
+ final UUID txId;
+ final VersionChain next;
+
+ VersionChain(BinaryRow row, Timestamp begin, UUID txId, VersionChain
next) {
+ this.row = row;
+ this.begin = begin;
+ this.txId = txId;
+ this.next = next;
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void addWrite(BinaryRow row, UUID txId) throws
TxIdMismatchException {
+ //TODO Make it idempotent?
+ map.compute(row.keySlice(), (keyBuf, versionChain) -> {
+ if (versionChain != null && versionChain.begin == null &&
!txId.equals(versionChain.txId)) {
+ throw new TxIdMismatchException();
+ }
+
+ return new VersionChain(row, null, txId, versionChain);
+ });
+
+ for (TestSortedIndexMvStorage index : indexes) {
+ index.append(row);
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void abortWrite(BinaryRow key) {
+ map.merge(key.keySlice(), NULL, (versionChain, ignored) -> {
+ assert versionChain != null;
+ assert versionChain.begin == null && versionChain.txId != null;
+
+ BinaryRow aborted = versionChain.row;
+
+ for (TestSortedIndexMvStorage index : indexes) {
+ abortWrite(versionChain.next, aborted, index);
+ }
+
+ return versionChain.next;
+ });
+ }
+
+ private void abortWrite(VersionChain head, BinaryRow aborted,
TestSortedIndexMvStorage index) {
+ for (VersionChain cur = head; cur != null; cur = cur.next) {
+ if (index.matches(aborted, cur.row)) {
+ return;
+ }
+ }
+
+ index.remove(aborted);
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void commitWrite(BinaryRow key, Timestamp timestamp) {
+ map.compute(key.keySlice(), (keyBuf, versionChain) -> {
+ assert versionChain != null;
Review comment:
This invariant must be guaranteed by external lock manager in the future
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
+
+ private final NavigableSet<BinaryRow> index;
+
+ private final SchemaDescriptor descriptor;
+ private final Map<Integer, TestMvPartitionStorage> pk;
+
+ private final int partitions;
+
+ private final IndexColumnView[] indexColumns;
+
+ private final int[] columnIndexes;
+
+ private final NativeType[] nativeTypes;
+
+ protected TestSortedIndexMvStorage(
+ String name,
+ TableView tableCfg,
+ SchemaDescriptor descriptor,
+ Map<Integer, TestMvPartitionStorage> pk
+ ) {
+ this.tableCfg = tableCfg;
+
+ this.descriptor = descriptor;
+
+ this.pk = pk;
+
+ partitions = tableCfg.partitions();
+
+ index = new ConcurrentSkipListSet<>((l, r) -> {
+ int cmp = compareColumns(l, r);
+
+ if (cmp != 0) {
+ return cmp;
+ }
+
+ return l.keySlice().compareTo(r.keySlice());
+ });
+
+ // Init columns.
+ NamedListView<? extends ColumnView> tblColumns = tableCfg.columns();
+
+ TableIndexView idxCfg = tableCfg.indices().get(name);
+
+ assert idxCfg instanceof SortedIndexView;
+
+ SortedIndexView sortedIdxCfg = (SortedIndexView) idxCfg;
+
+ NamedListView<? extends IndexColumnView> columns =
sortedIdxCfg.columns();
+
+ int length = columns.size();
+
+ this.indexColumns = new IndexColumnView[length];
+ this.columnIndexes = new int[length];
+ this.nativeTypes = new NativeType[length];
+
+ for (int i = 0; i < length; i++) {
+ IndexColumnView idxColumn = columns.get(i);
+
+ indexColumns[i] = idxColumn;
+
+ int columnIndex =
tblColumns.namedListKeys().indexOf(idxColumn.name());
+
+ columnIndexes[i] = columnIndex;
+
+ nativeTypes[i] =
SchemaDescriptorConverter.convert(SchemaConfigurationConverter.convert(tblColumns.get(columnIndex).type()));
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean supportsBackwardsScan() {
+ return true;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean supportsIndexOnlyScan() {
+ return false;
+ }
+
+ private int compareColumns(BinaryRow l, BinaryRow r) {
+ Row leftRow = new Row(descriptor, l);
+ Row rightRow = new Row(descriptor, r);
+
+ for (int i = 0; i < indexColumns.length; i++) {
+ int columnIndex = columnIndexes[i];
+
+ int cmp = PrefixComparator.compareColumns(leftRow, columnIndex,
nativeTypes[i].spec(), rightRow.value(columnIndex));
+
+ if (cmp != 0) {
+ return indexColumns[i].asc() ? cmp : -cmp;
+ }
+ }
+
+ return 0;
+ }
+
+ public void append(BinaryRow row) {
+ index.add(row);
+ }
+
+ public void remove(BinaryRow row) {
+ index.remove(row);
+ }
+
+ public boolean matches(BinaryRow aborted, BinaryRow existing) {
+ return compareColumns(aborted, existing) == 0;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public Cursor<IndexRowEx> scan(
+ @Nullable IndexRowPrefix lowerBound,
+ @Nullable IndexRowPrefix upperBound,
+ int flags,
+ Timestamp timestamp,
+ @Nullable IntPredicate partitionFilter
+ ) {
+ boolean includeLower = (flags & GREATER_OR_EQUAL) != 0;
+ boolean includeUpper = (flags & LESS_OR_EQUAL) != 0;
+
+ NavigableSet<BinaryRow> index = this.index;
+ int direction = 1;
+
+ // Swap bounds and flip index for backwards scan.
+ if ((flags & BACKWARDS) != 0) {
+ index = index.descendingSet();
+ direction = -1;
+
+ boolean tempBoolean = includeLower;
+ includeLower = includeUpper;
+ includeUpper = tempBoolean;
+
+ IndexRowPrefix tempBound = lowerBound;
+ lowerBound = upperBound;
+ upperBound = tempBound;
+ }
+
+ ToIntFunction<BinaryRow> lowerCmp = lowerBound == null ? row -> 1 :
boundComparator(lowerBound, direction, includeLower ? 0 : -1);
+ ToIntFunction<BinaryRow> upperCmp = upperBound == null ? row -> -1 :
boundComparator(upperBound, direction, includeUpper ? 0 : 1);
+
+ Iterator<IndexRowEx> iterator = index.stream()
+ .dropWhile(binaryRow -> {
+ return lowerCmp.applyAsInt(binaryRow) < 0;
Review comment:
Ok
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
+
+ private final NavigableSet<BinaryRow> index;
+
+ private final SchemaDescriptor descriptor;
+ private final Map<Integer, TestMvPartitionStorage> pk;
+
+ private final int partitions;
+
+ private final IndexColumnView[] indexColumns;
+
+ private final int[] columnIndexes;
+
+ private final NativeType[] nativeTypes;
+
+ protected TestSortedIndexMvStorage(
+ String name,
+ TableView tableCfg,
+ SchemaDescriptor descriptor,
+ Map<Integer, TestMvPartitionStorage> pk
+ ) {
+ this.tableCfg = tableCfg;
+
+ this.descriptor = descriptor;
+
+ this.pk = pk;
+
+ partitions = tableCfg.partitions();
+
+ index = new ConcurrentSkipListSet<>((l, r) -> {
+ int cmp = compareColumns(l, r);
+
+ if (cmp != 0) {
+ return cmp;
+ }
+
+ return l.keySlice().compareTo(r.keySlice());
+ });
+
+ // Init columns.
+ NamedListView<? extends ColumnView> tblColumns = tableCfg.columns();
+
+ TableIndexView idxCfg = tableCfg.indices().get(name);
+
+ assert idxCfg instanceof SortedIndexView;
+
+ SortedIndexView sortedIdxCfg = (SortedIndexView) idxCfg;
+
+ NamedListView<? extends IndexColumnView> columns =
sortedIdxCfg.columns();
+
+ int length = columns.size();
+
+ this.indexColumns = new IndexColumnView[length];
+ this.columnIndexes = new int[length];
+ this.nativeTypes = new NativeType[length];
+
+ for (int i = 0; i < length; i++) {
+ IndexColumnView idxColumn = columns.get(i);
+
+ indexColumns[i] = idxColumn;
+
+ int columnIndex =
tblColumns.namedListKeys().indexOf(idxColumn.name());
+
+ columnIndexes[i] = columnIndex;
+
+ nativeTypes[i] =
SchemaDescriptorConverter.convert(SchemaConfigurationConverter.convert(tblColumns.get(columnIndex).type()));
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean supportsBackwardsScan() {
+ return true;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean supportsIndexOnlyScan() {
+ return false;
+ }
+
+ private int compareColumns(BinaryRow l, BinaryRow r) {
+ Row leftRow = new Row(descriptor, l);
Review comment:
It's weird that we have two almost identical comparison methods
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
+
+ private final NavigableSet<BinaryRow> index;
+
+ private final SchemaDescriptor descriptor;
+ private final Map<Integer, TestMvPartitionStorage> pk;
+
+ private final int partitions;
+
+ private final IndexColumnView[] indexColumns;
+
+ private final int[] columnIndexes;
+
+ private final NativeType[] nativeTypes;
+
+ protected TestSortedIndexMvStorage(
+ String name,
+ TableView tableCfg,
+ SchemaDescriptor descriptor,
+ Map<Integer, TestMvPartitionStorage> pk
+ ) {
+ this.tableCfg = tableCfg;
+
+ this.descriptor = descriptor;
+
+ this.pk = pk;
+
+ partitions = tableCfg.partitions();
+
+ index = new ConcurrentSkipListSet<>((l, r) -> {
+ int cmp = compareColumns(l, r);
+
+ if (cmp != 0) {
+ return cmp;
+ }
+
+ return l.keySlice().compareTo(r.keySlice());
+ });
+
+ // Init columns.
+ NamedListView<? extends ColumnView> tblColumns = tableCfg.columns();
+
+ TableIndexView idxCfg = tableCfg.indices().get(name);
+
+ assert idxCfg instanceof SortedIndexView;
+
+ SortedIndexView sortedIdxCfg = (SortedIndexView) idxCfg;
+
+ NamedListView<? extends IndexColumnView> columns =
sortedIdxCfg.columns();
+
+ int length = columns.size();
+
+ this.indexColumns = new IndexColumnView[length];
+ this.columnIndexes = new int[length];
+ this.nativeTypes = new NativeType[length];
+
+ for (int i = 0; i < length; i++) {
+ IndexColumnView idxColumn = columns.get(i);
+
+ indexColumns[i] = idxColumn;
+
+ int columnIndex =
tblColumns.namedListKeys().indexOf(idxColumn.name());
+
+ columnIndexes[i] = columnIndex;
+
+ nativeTypes[i] =
SchemaDescriptorConverter.convert(SchemaConfigurationConverter.convert(tblColumns.get(columnIndex).type()));
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean supportsBackwardsScan() {
+ return true;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public boolean supportsIndexOnlyScan() {
+ return false;
+ }
+
+ private int compareColumns(BinaryRow l, BinaryRow r) {
+ Row leftRow = new Row(descriptor, l);
Review comment:
Again, there's a mismatch with columns indexes, as far as I know
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage;
+
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Predicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.apache.ignite.lang.IgniteException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Multi-versioned partition storage.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Their API are very similar,
+ * althouh there are very important differences that will be addressed in the
future.
+ */
+public interface MvPartitionStorage {
+ /**
+ * Exception class that describes the situation where two independant
transactions attempting to write values for the same key.
+ */
+ class TxIdMismatchException extends IgniteException {
+ }
+
+ /**
+ * Reads the value from the storage as it was at the given timestamp.
Review comment:
Ok
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/PartitionStorage.java
##########
@@ -128,6 +143,66 @@
*/
Cursor<DataRow> scan(Predicate<SearchRow> filter) throws StorageException;
+ /**
+ * Scans the partition and returns a cursor of values in at the given
timestamp.
+ *
+ * @param keyFilter Key filter. Binary rows passed to the filter may or
may not have a value, filter should only check keys.
+ * @param timestamp Timestamp
+ * @return Cursor.
+ */
+ default Cursor<BinaryRow> scan(Predicate<BinaryRow> keyFilter, @Nullable
Timestamp timestamp) {
+ throw new UnsupportedOperationException("scan");
+ }
+
+ /**
+ * Exception class that describes the situation where two independant
transactions attempting to write values for the same key.
+ */
+ class TxIdMismatchException extends RuntimeException {
+ }
Review comment:
All inner classes in interfaces are implicitly static
##########
File path:
modules/core/src/main/java/org/apache/ignite/internal/util/Cursor.java
##########
@@ -25,5 +25,38 @@
* @param <T> Type of elements.
*/
public interface Cursor<T> extends Iterator<T>, Iterable<T>, AutoCloseable {
+ /**
+ * Creates an iterator based cursor.
+ *
+ * @param it Iterator.
+ * @param <T> Type of elements in iterator.
+ * @return Cursor.
+ */
+ static <T> Cursor<T> fromIterator(Iterator<? extends T> it) {
+ return new Cursor<T>() {
+ /** {@inheritDoc} */
+ @Override
+ public void close() throws Exception {
+ // No-op.
+ }
+ /** {@inheritDoc} */
+ @Override
+ public Iterator<T> iterator() {
+ return this;
+ }
Review comment:
I'll do it, thank you!
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/PartitionStorage.java
##########
@@ -128,6 +143,66 @@
*/
Cursor<DataRow> scan(Predicate<SearchRow> filter) throws StorageException;
+ /**
+ * Scans the partition and returns a cursor of values in at the given
timestamp.
+ *
+ * @param keyFilter Key filter. Binary rows passed to the filter may or
may not have a value, filter should only check keys.
+ * @param timestamp Timestamp
+ * @return Cursor.
+ */
+ default Cursor<BinaryRow> scan(Predicate<BinaryRow> keyFilter, @Nullable
Timestamp timestamp) {
+ throw new UnsupportedOperationException("scan");
+ }
+
+ /**
+ * Exception class that describes the situation where two independant
transactions attempting to write values for the same key.
+ */
+ class TxIdMismatchException extends RuntimeException {
+ }
Review comment:
My mistake
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/index/SortedIndexMvStorage.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.index;
+
+import java.util.function.IntPredicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.intellij.lang.annotations.MagicConstant;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Storage for a sorted index.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Real future implementation
+ * will be defined later. Things to notice here: TX-aware implementation
should have a projection bitset instead of full row reading.
+ * Physical storage API will be enriched with append/remove methods, like in
reference implementation.
+ */
+public interface SortedIndexMvStorage {
+ /** Exclude lower bound. */
+ int GREATER = 0;
Review comment:
These are default values, I didn't use them in tests explicitly, maybe I
should fix that
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/index/SortedIndexMvStorage.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.index;
+
+import java.util.function.IntPredicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.intellij.lang.annotations.MagicConstant;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Storage for a sorted index.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Real future implementation
+ * will be defined later. Things to notice here: TX-aware implementation
should have a projection bitset instead of full row reading.
+ * Physical storage API will be enriched with append/remove methods, like in
reference implementation.
+ */
+public interface SortedIndexMvStorage {
+ /** Exclude lower bound. */
+ int GREATER = 0;
+
+ /** Include lower bound. */
+ int GREATER_OR_EQUAL = 1;
+
+ /** Exclude upper bound. */
+ int LESS = 0;
+
+ /** Include upper bound. */
+ int LESS_OR_EQUAL = 1 << 1;
+
+ /** Forward scan. */
+ int FORWARD = 0;
+
+ /** Backwards scan. */
+ int BACKWARDS = 1 << 2;
+
+ /**
+ * The sole purpose of this class is to avoid massive refactoring while
changing the original IndexRow.
+ */
+ interface IndexRowEx {
+ /**
+ * Key-only binary row if index-only scan is supported, full binary
row otherwise.
+ */
+ BinaryRow row();
+
+ /**
+ * Returns indexed column value.
+ *
+ * @param idx PK column index.
+ * @return Indexed column value.
+ */
+ Object value(int idx);
+ }
+
+ boolean supportsBackwardsScan();
+
+ boolean supportsIndexOnlyScan();
+
+ /**
+ * Returns a range of index values between the lower bound and the upper
bound, consistent with the passed timestamp.
Review comment:
It means the state of the data as it was at the given time
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestMvPartitionStorage.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Predicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.storage.MvPartitionStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV partition storage.
+ */
+public class TestMvPartitionStorage implements MvPartitionStorage {
+ private static final VersionChain NULL = new VersionChain(null, null,
null, null);
+
+ private final ConcurrentHashMap<ByteBuffer, VersionChain> map = new
ConcurrentHashMap<>();
+
+ private final List<TestSortedIndexMvStorage> indexes;
+
+ public TestMvPartitionStorage(List<TestSortedIndexMvStorage> indexes) {
+ this.indexes = indexes;
+ }
+
+ private static class VersionChain {
+ final BinaryRow row;
+ final Timestamp begin;
+ final UUID txId;
+ final VersionChain next;
+
+ VersionChain(BinaryRow row, Timestamp begin, UUID txId, VersionChain
next) {
Review comment:
Ok
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage;
+
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Predicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.apache.ignite.lang.IgniteException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Multi-versioned partition storage.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Their API are very similar,
+ * althouh there are very important differences that will be addressed in the
future.
+ */
+public interface MvPartitionStorage {
+ /**
+ * Exception class that describes the situation where two independant
transactions attempting to write values for the same key.
+ */
+ class TxIdMismatchException extends IgniteException {
+ }
+
+ /**
+ * Reads the value from the storage as it was at the given timestamp.
+ *
+ * @param key Key.
+ * @param timestamp Timestamp.
+ * @return Binary row that corresponds to the key or {@code null} if value
is not found.
+ */
+ @Nullable
+ BinaryRow read(BinaryRow key, @Nullable Timestamp timestamp);
+
+ /**
+ * Creates uncommited version, assigned to the passed transaction id..
+ *
+ * @param row Binary row to update. Key only row means value removal.
+ * @param txId Transaction id.
+ * @throws TxIdMismatchException If there's another pending update
associated with different transaction id.
+ * @throws StorageException If failed to write data to the storage.
+ */
+ void addWrite(BinaryRow row, UUID txId) throws TxIdMismatchException,
StorageException;
+
+ /**
+ * Aborts a pending update of the ongoing uncommited transaction. Invoked
during rollback.
+ *
+ * @param key Key.
+ * @throws StorageException If failed to write data to the storage.
+ */
+ void abortWrite(BinaryRow key) throws StorageException;
+
+ /**
+ * Commits a pending update of the ongoing transaction. Invoked during
commit. Commited value will be versioned by the given timestamp.
+ *
+ * @param key Key.
+ * @param timestamp Timestamp to associate with commited value.
+ * @throws StorageException If failed to write data to the storage.
+ */
+ void commitWrite(BinaryRow key, Timestamp timestamp) throws
StorageException;
+
+ /**
+ * Removes data associated with old timestamps.
+ *
+ * @param from Start of hashes range to process. Inclusive.
+ * @param to End of hashes range to process. Inclusive.
+ * @param timestamp Timestamp to remove all the data with a lesser
timestamp.
+ * @return Future for the operation.
+ */
+ CompletableFuture<?> cleanup(int from, int to, Timestamp timestamp);
Review comment:
That's right, it's neither implemented nor ever used
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
+
+ private final NavigableSet<BinaryRow> index;
+
+ private final SchemaDescriptor descriptor;
+ private final Map<Integer, TestMvPartitionStorage> pk;
+
+ private final int partitions;
+
+ private final IndexColumnView[] indexColumns;
+
+ private final int[] columnIndexes;
+
+ private final NativeType[] nativeTypes;
+
+ protected TestSortedIndexMvStorage(
Review comment:
There are no reasons for this
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/index/SortedIndexMvStorage.java
##########
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.index;
+
+import java.util.function.IntPredicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.intellij.lang.annotations.MagicConstant;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Storage for a sorted index.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Real future implementation
+ * will be defined later. Things to notice here: TX-aware implementation
should have a projection bitset instead of full row reading.
+ * Physical storage API will be enriched with append/remove methods, like in
reference implementation.
+ */
+public interface SortedIndexMvStorage {
+ /** Exclude lower bound. */
+ int GREATER = 0;
+
+ /** Include lower bound. */
+ int GREATER_OR_EQUAL = 1;
+
+ /** Exclude upper bound. */
+ int LESS = 0;
+
+ /** Include upper bound. */
+ int LESS_OR_EQUAL = 1 << 1;
+
+ /** Forward scan. */
+ int FORWARD = 0;
+
+ /** Backwards scan. */
+ int BACKWARDS = 1 << 2;
+
+ /**
+ * The sole purpose of this class is to avoid massive refactoring while
changing the original IndexRow.
+ */
+ interface IndexRowEx {
+ /**
+ * Key-only binary row if index-only scan is supported, full binary
row otherwise.
+ */
+ BinaryRow row();
+
+ /**
+ * Returns indexed column value.
+ *
+ * @param idx PK column index.
+ * @return Indexed column value.
+ */
+ Object value(int idx);
+ }
+
+ boolean supportsBackwardsScan();
Review comment:
Guys asked for them, there's a chance that we won't support backwards
scan in B+Tree, for example, but who knows
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage;
+
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Predicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.apache.ignite.lang.IgniteException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Multi-versioned partition storage.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Their API are very similar,
+ * althouh there are very important differences that will be addressed in the
future.
+ */
+public interface MvPartitionStorage {
+ /**
+ * Exception class that describes the situation where two independant
transactions attempting to write values for the same key.
+ */
+ class TxIdMismatchException extends IgniteException {
Review comment:
Sure
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/BaseMvStoragesTest.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage;
+
+import java.util.Locale;
+import java.util.Objects;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.Column;
+import org.apache.ignite.internal.schema.NativeTypes;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import org.apache.ignite.internal.schema.marshaller.KvMarshaller;
+import org.apache.ignite.internal.schema.marshaller.MarshallerException;
+import org.apache.ignite.internal.schema.marshaller.MarshallerFactory;
+import
org.apache.ignite.internal.schema.marshaller.reflection.ReflectionMarshallerFactory;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.lang.IgniteException;
+import org.jetbrains.annotations.Nullable;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.BeforeAll;
+
+/**
+ * Base test for MV storages, contains pojo classes, their descriptor and a
marshaller instance.
+ */
+public class BaseMvStoragesTest {
+ /** Default reflection marshaller factory. */
+ protected static MarshallerFactory marshallerFactory;
+
+ /** Schema descriptor for tests. */
+ protected static SchemaDescriptor schemaDescriptor;
+
+ /** Key-value marshaller for tests. */
+ protected static KvMarshaller<TestKey, TestValue> kvMarshaller;
+
+ @BeforeAll
+ static void beforeAll() {
+ marshallerFactory = new ReflectionMarshallerFactory();
+
+ schemaDescriptor = new SchemaDescriptor(1, new Column[]{
+ new Column("intKey".toUpperCase(Locale.ROOT),
NativeTypes.INT32, false),
+ new Column("strKey".toUpperCase(Locale.ROOT),
NativeTypes.STRING, false),
+ }, new Column[]{
+ new Column("intVal".toUpperCase(Locale.ROOT),
NativeTypes.INT32, false),
+ new Column("strVal".toUpperCase(Locale.ROOT),
NativeTypes.STRING, false),
+ });
+
+ kvMarshaller = marshallerFactory.create(schemaDescriptor,
TestKey.class, TestValue.class);
+ }
+
+ @AfterAll
+ static void afterAll() {
+ kvMarshaller = null;
+ schemaDescriptor = null;
+ marshallerFactory = null;
+ }
+
+ protected BinaryRow binaryKey(TestKey key) {
Review comment:
Correct, I missed that somehow
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
+
+ private final NavigableSet<BinaryRow> index;
+
+ private final SchemaDescriptor descriptor;
+ private final Map<Integer, TestMvPartitionStorage> pk;
+
+ private final int partitions;
+
+ private final IndexColumnView[] indexColumns;
+
+ private final int[] columnIndexes;
+
+ private final NativeType[] nativeTypes;
+
+ protected TestSortedIndexMvStorage(
+ String name,
+ TableView tableCfg,
+ SchemaDescriptor descriptor,
+ Map<Integer, TestMvPartitionStorage> pk
+ ) {
+ this.tableCfg = tableCfg;
+
+ this.descriptor = descriptor;
+
+ this.pk = pk;
+
+ partitions = tableCfg.partitions();
+
+ index = new ConcurrentSkipListSet<>((l, r) -> {
+ int cmp = compareColumns(l, r);
+
+ if (cmp != 0) {
+ return cmp;
+ }
+
+ return l.keySlice().compareTo(r.keySlice());
+ });
+
+ // Init columns.
+ NamedListView<? extends ColumnView> tblColumns = tableCfg.columns();
+
+ TableIndexView idxCfg = tableCfg.indices().get(name);
+
+ assert idxCfg instanceof SortedIndexView;
+
+ SortedIndexView sortedIdxCfg = (SortedIndexView) idxCfg;
+
+ NamedListView<? extends IndexColumnView> columns =
sortedIdxCfg.columns();
+
+ int length = columns.size();
+
+ this.indexColumns = new IndexColumnView[length];
Review comment:
There must have been a reason for this, I think SortedIndexDescriptor
uses different indexes for columns. This code addresses the original BinaryRow,
not the "compressed" one that only has the required columns
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
Review comment:
Indeed
##########
File path:
modules/storage-api/src/test/java/org/apache/ignite/internal/storage/basic/TestSortedIndexMvStorage.java
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage.basic;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentSkipListSet;
+import java.util.function.IntPredicate;
+import java.util.function.ToIntFunction;
+import org.apache.ignite.configuration.NamedListView;
+import org.apache.ignite.configuration.schemas.table.ColumnView;
+import org.apache.ignite.configuration.schemas.table.IndexColumnView;
+import org.apache.ignite.configuration.schemas.table.SortedIndexView;
+import org.apache.ignite.configuration.schemas.table.TableIndexView;
+import org.apache.ignite.configuration.schemas.table.TableView;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.schema.NativeType;
+import org.apache.ignite.internal.schema.SchemaDescriptor;
+import
org.apache.ignite.internal.schema.configuration.SchemaConfigurationConverter;
+import
org.apache.ignite.internal.schema.configuration.SchemaDescriptorConverter;
+import org.apache.ignite.internal.schema.row.Row;
+import org.apache.ignite.internal.storage.index.IndexRowPrefix;
+import org.apache.ignite.internal.storage.index.PrefixComparator;
+import org.apache.ignite.internal.storage.index.SortedIndexMvStorage;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Test implementation of MV sorted index storage.
+ */
+public class TestSortedIndexMvStorage implements SortedIndexMvStorage {
+ private final TableView tableCfg;
+
+ private final NavigableSet<BinaryRow> index;
+
+ private final SchemaDescriptor descriptor;
+ private final Map<Integer, TestMvPartitionStorage> pk;
+
+ private final int partitions;
+
+ private final IndexColumnView[] indexColumns;
+
+ private final int[] columnIndexes;
+
+ private final NativeType[] nativeTypes;
+
+ protected TestSortedIndexMvStorage(
+ String name,
+ TableView tableCfg,
+ SchemaDescriptor descriptor,
+ Map<Integer, TestMvPartitionStorage> pk
+ ) {
+ this.tableCfg = tableCfg;
+
+ this.descriptor = descriptor;
+
+ this.pk = pk;
+
+ partitions = tableCfg.partitions();
+
+ index = new ConcurrentSkipListSet<>((l, r) -> {
+ int cmp = compareColumns(l, r);
Review comment:
Thank you
##########
File path:
modules/storage-api/src/main/java/org/apache/ignite/internal/storage/MvPartitionStorage.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.storage;
+
+import java.util.UUID;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Predicate;
+import org.apache.ignite.internal.schema.BinaryRow;
+import org.apache.ignite.internal.tx.Timestamp;
+import org.apache.ignite.internal.util.Cursor;
+import org.apache.ignite.lang.IgniteException;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Multi-versioned partition storage.
+ * POC version, that represents a combination between a replicated TX-aware MV
storage and physical MV storage. Their API are very similar,
+ * althouh there are very important differences that will be addressed in the
future.
+ */
+public interface MvPartitionStorage {
+ /**
+ * Exception class that describes the situation where two independant
transactions attempting to write values for the same key.
+ */
+ class TxIdMismatchException extends IgniteException {
+ }
+
+ /**
+ * Reads the value from the storage as it was at the given timestamp.
+ *
+ * @param key Key.
+ * @param timestamp Timestamp.
+ * @return Binary row that corresponds to the key or {@code null} if value
is not found.
+ */
+ @Nullable
+ BinaryRow read(BinaryRow key, @Nullable Timestamp timestamp);
+
+ /**
+ * Creates uncommited version, assigned to the passed transaction id..
+ *
+ * @param row Binary row to update. Key only row means value removal.
+ * @param txId Transaction id.
+ * @throws TxIdMismatchException If there's another pending update
associated with different transaction id.
+ * @throws StorageException If failed to write data to the storage.
+ */
+ void addWrite(BinaryRow row, UUID txId) throws TxIdMismatchException,
StorageException;
+
+ /**
+ * Aborts a pending update of the ongoing uncommited transaction. Invoked
during rollback.
+ *
+ * @param key Key.
+ * @throws StorageException If failed to write data to the storage.
+ */
+ void abortWrite(BinaryRow key) throws StorageException;
+
+ /**
+ * Commits a pending update of the ongoing transaction. Invoked during
commit. Commited value will be versioned by the given timestamp.
+ *
+ * @param key Key.
+ * @param timestamp Timestamp to associate with commited value.
+ * @throws StorageException If failed to write data to the storage.
+ */
+ void commitWrite(BinaryRow key, Timestamp timestamp) throws
StorageException;
+
+ /**
+ * Removes data associated with old timestamps.
+ *
+ * @param from Start of hashes range to process. Inclusive.
Review comment:
You know what, I'll drop this method completely. Things change too
quickly, this signature makes no sense most likely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
