Luo Chen has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/1813
Change subject: Add Create Secondary BTree for Correlated Datasets ...................................................................... Add Create Secondary BTree for Correlated Datasets Implemented create seconary BTree index for datasets using correlated merge policy. Instead of creating one component for the new index, this change creates one component for each component of the primary index to maintain the correlation. The current implementation assumes when a secondary index is being created, the dataset is locked with no modifications. Change-Id: I2a3435e6720f07bd6a5092d4d9ce42e8d4b7894c --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.1.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.2.update.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.3.ddl.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries/upsert/primary-correlated-secondary-btree/primary-correlated-secondary-btree.4.query.aql A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-nullable/scan-delete-btree-correlated-secondary-index-nullable.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-delete-btree-correlated-secondary-index-open/scan-delete-btree-correlated-secondary-index-open.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-nullable/scan-insert-btree-correlated-secondary-index-nullable.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/scan-insert-btree-correlated-secondary-index-open/scan-insert-btree-correlated-secondary-index-open.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/dml/using-correlated-prefix-merge-policy-with-feed/using-correlated-prefix-merge-policy-with-feed.5.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/filters/insert-with-correlated-secondary-btree/insert-with-correlated-secondary-btree.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/temp-dataset/insert-and-scan-dataset-with-correlated-index/insert-and-scan-dataset-with-correlated-index.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.2.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/upsert/primary-correlated-secondary-btree/primary-secondary-btree.4.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.1.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.10.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.2.ddl.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.3.update.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.4.txneu.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.5.ddl.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.6.txnqbc.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.7.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.8.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/recover_after_abort/primary_plus_default_correlated_secondary_index/primary_plus_default_correlated_secondary_index.9.txnqar.aql A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/create_and_start.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/kill_cc_and_nc.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/stop_and_delete.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/recover_after_abort/primary_plus_default_correlated_secondary_index/stop_and_start.sh M asterixdb/asterix-installer/src/test/resources/transactionts/testsuite.xml M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Dataset.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/DatasetUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/IndexUtil.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryCorrelatedBTreeOperationsHelper.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryCorrelatedTreeIndexOperationsHelper.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SecondaryIndexOperationsHelper.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/AbstractLSMSecondaryCreationNodePushable.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryCreationBulkLoadNodePushable.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryCreationBulkLoadOperatorDescriptor.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryCreationTupleProcessorNodePushable.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/LSMSecondaryCreationTupleProcessorOperatorDescriptor.java A asterixdb/asterix-runtime/src/test/java/org/apache/asterix/runtime/operators/LSMSecondaryCreationTupleProcessorTest.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/pom.xml A hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeDiskComponentScanOperatorDescriptor.java A hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/dataflow/LSMBTreeDiskComponentScanOperatorNodePushable.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/ExternalBTreeWithBuddy.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-btree/src/main/java/org/apache/hyracks/storage/am/lsm/btree/impls/LSMBTreeDiskComponentScanCursor.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/api/ILSMIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-invertedindex/src/main/java/org/apache/hyracks/storage/am/lsm/invertedindex/impls/LSMInvertedIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-rtree/src/main/java/org/apache/hyracks/storage/am/lsm/rtree/impls/LSMRTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-rtree/src/main/java/org/apache/hyracks/storage/am/lsm/rtree/impls/LSMRTreeWithAntiMatterTuples.java 105 files changed, 4,704 insertions(+), 103 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/13/1813/1 -- To view, visit https://asterix-gerrit.ics.uci.edu/1813 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2a3435e6720f07bd6a5092d4d9ce42e8d4b7894c Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Luo Chen <[email protected]>
