Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2: Integration-Tests-1

Integration Tests Failed

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4717/ 
: UNSTABLE

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17: Integration-Tests-1

Integration Tests Failed

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4715/ 
: UNSTABLE

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4717/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1699/ 
(11/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-rebalance-cancellation/888/
 (10/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/7239/ (6/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2315/ (3/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1708/
 (1/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Murtadha Hubail (Code Review)]

Murtadha Hubail has uploaded a new patch set (#2).

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..

[NO ISSUE][STO] Adapt Storage Structure To Rebalance

- user model changes: no
- storage format changes: no
- interface changes: yes
-- Added IResource#setPath to use for the resource
   storage migration.

Details:
- Unify storage structure to support dataset rebalance:
  Old format:
  ./storage/partition_#/dataverse/datasetName_idx_indexName
  New format:
  ./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName
- Adapt recovery and replication to new storage structure.
- Add old structure -> new structure NC migration task.
- Add CompatibilityUtil to ensure NC can be upgraded during
  NC startup.
- Centralize the logic for parsing file path to its components in
  ResourceReference/DatasetResourceReference.
- Add storage structure migration test case.
- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/MigrateStorageResourcesTask.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/CompatibilityUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/common/TestDataUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/MigrateStorageResourcesTaskTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/txn/RecoveryManagerTest.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/DatasetLocalResource.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/DatasetResourceReference.java
D 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexFileProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexPathElements.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ResourceReference.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterixdb/asterix-replication/pom.xml
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMComponentProperties.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/dataflow/LsmResource.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-rtree/src/main/java/org/apache/hyracks/storage/am/rtree/dataflow/RTreeResource.java
M 
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/IResource.java
28 files changed, 882 insertions(+), 422 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/81/2181/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/2563/ (5/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/2373/ 
(9/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/2352/
 (7/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/2018/ 
(2/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1687/ 
(8/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 2:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5739/ (4/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storage Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Storage Structure To Rebalance
..


Patch Set 1:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/2017/ 
(10/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/7238/ (5/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2314/ (2/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-rebalance-cancellation/887/
 (8/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1698/ 
(7/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/2372/ 
(9/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1686/ 
(3/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5738/ (11/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/2562/ (4/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/2351/
 (6/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Murtadha Hubail (Code Review)]

Murtadha Hubail has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/2181

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..

[NO ISSUE][STO] Adapt Structure Structure To Rebalance

- user model changes: no
- storage format changes: no
- interface changes: yes
-- Added IResource#setPath to use for the resource
   storage migration.

Details:
- Unify storage structure to support dataset rebalance:
  Old format:
  ./storage/partition_#/dataverse/datasetName_idx_indexName
  New format:
  ./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName
- Adapt recovery and replication to new storage structure.
- Add old structure -> new structure NC migration task.
- Add CompatibilityUtil to ensure NC can be upgraded during
  NC startup.
- Centralize the logic for parsing file path to its components in
  ResourceReference/DatasetResourceReference.
- Add storage structure migration test case.
- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/MigrateStorageResourcesTask.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/CompatibilityUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/common/TestDataUtil.java
A 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/MigrateStorageResourcesTaskTest.java
M 
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/txn/RecoveryManagerTest.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/DatasetLocalResource.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/DatasetResourceReference.java
D 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexFileProperties.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexPathElements.java
A 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ResourceReference.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterixdb/asterix-replication/pom.xml
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMComponentProperties.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M 
asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M 
asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/dataflow/LsmResource.java
M 
hyracks-fullstack/hyracks/hyracks-storage-am-rtree/src/main/java/org/apache/hyracks/storage/am/rtree/dataflow/RTreeResource.java
M 
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/IResource.java
28 files changed, 882 insertions(+), 422 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/81/2181/1

diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
index 19966fe..e29e3fe 100644
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
@@ -443,7 +443,7 @@
 return minFirstLSN;
 }
 
-private long getRemoteMinFirstLSN() {
+private long getRemoteMinFirstLSN() throws HyracksDataException {
 IReplicaResourcesManager remoteResourcesManager =
 
txnSubsystem.getAsterixAppRuntimeContextProvider().getAppContext().getReplicaResourcesManager();
 return 

Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance
..


Patch Set 1:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1707/
 (1/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail 
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

WARNING: THIS CHANGE CONTAINS CROSS-PRODUCT CHANGES IN:
* asterixdb
* hyracks-fullstack

PLEASE REVIEW CAREFULLY AND LOOK FOR API CHANGES!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Integration Tests Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/4715/

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-sonar/5737/ (10/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app-sql-execution/1706/
 (1/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-rebalance-cancellation/886/
 (6/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/1697/ 
(5/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started https://asterix-jenkins.ics.uci.edu/job/hyracks-gerrit/1685/ 
(3/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-asterix-app/2371/ 
(7/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/2016/ 
(8/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-verify-no-installer-app/2350/
 (11/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-asterix-app/2561/ (9/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-verify-storage/2313/ (2/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Taewoo Kim (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1866

to look at the new patch set (#17).

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..

[ASTERIXDB-1972][COMP][RT][TX] index-only plan

- user model changes: no
- storage format changes: no
- interface changes: IAccessMethod, ILSMIndexOperationContext,
 IIndexAccessor

Details:
- Implement an index-only plan

Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/BTreeSearchPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/RTreeSearchPOperator.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AbstractIntroduceAccessMethodRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AccessMethodAnalysisContext.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/AccessMethodUtils.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/BTreeAccessMethod.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IAccessMethod.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceJoinAccessMethodRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceLSMComponentFilterRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroducePrimaryIndexForAggregationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/IntroduceSelectAccessMethodRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/InvertedIndexAccessMethod.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/OptimizableOperatorSubTree.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/RTreeAccessMethod.java
A asterixdb/asterix-app/data/nontagged/customerData2.json
M 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/leftouterjoin-probe-pidx-with-join-btree-sidx_01.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/primary-to-secondary-indexonly-plan-equi-join_01.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-equi-join_04.sqlpp
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-equi-join_05.sqlpp
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-equi-join_06.sqlpp
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-indexonly-plan-to-primary-equi-join_01.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-indexonly-plan-to-secondary-indexonly-plan-equi-join_01.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-non-indexonly-plan-to-secondary-indexonly-plan-equi-join_01.aql
M 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index-join/secondary-self-equi-join.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-composite-index-indexonly-plan-01.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-01.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-02.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-03.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-04.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-05.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-06.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-07.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/btree-index/btree-secondary-index-indexonly-plan-08.aql
M 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/rtree-index-join/spatial-intersect-point_02.aql
M 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/rtree-index-join/spatial-intersect-point_03.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/rtree-index-join/spatial-intersect-point_04.aql
A 
asterixdb/asterix-app/src/test/resources/optimizerts/queries/rtree-index-join/spatial-intersect-point_05.aql
A 

Change in asterixdb[master]: [ASTERIXDB-1972][COMP][RT][TX] index-only plan

[Jenkins (Code Review)]

Jenkins has posted comments on this change.

Change subject: [ASTERIXDB-1972][COMP][RT][TX] index-only plan
..


Patch Set 17:

Build Started 
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/7237/ (4/11)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1866
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ifd5c9ab1cf2e4bedb7d8db582441919875e74d51
Gerrit-PatchSet: 17
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim 
Gerrit-Reviewer: Dmitry Lychagin 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-HasComments: No

[jira] [Commented] (ASTERIXDB-2176) Deletion doesn't work on the RTree index.

[Taewoo Kim (JIRA)]

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265520#comment-16265520
 ] 

Taewoo Kim commented on ASTERIXDB-2176:
---

Adding one more point:

A sqlpp execution test case: "scan-delete-rtree-secondary-index" conducts the 
following:

(1) creates a dataset 
(2) loads 21 records. (id = 1 ~ 21)
(3) creates a RTree index
(4) deletes 10 records (id > 10) 
(5) query

On the master branch, an RTree search reveals id: 12 and 20. These records are 
filtered by the primary-index search since it cannot find the record 12 and 20. 
So it's not seen. Overall, I think deletion on RTree doesn't work now. And it 
is hidden because of the primary-index search.

> Deletion doesn't work on the RTree index.
> -
>
> Key: ASTERIXDB-2176
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2176
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Taewoo Kim
>
> This is a simplified version of "upsert/primary-secondary-tree" AQL test case.
> spatialData.json file
> { "id": 12, "point": point("6.0,3.0") }
> moreSpatialData.json file
> {"id": 12, "point": point("4.1,7.0")}
> DDL: 
> {code}
> drop dataverse test if exists;
> create dataverse test;
> use dataverse test;
> create type MyRecord as closed {
>   id: int64,
>   point: point
> }
> create dataset UpsertTo(MyRecord)
>  primary key id;
> create dataset UpsertFrom(MyRecord)
>  primary key id;
> create index rtree_index_point on UpsertTo(point) type rtree;
> {code}
> DML
> {code}
> load dataset UpsertTo
> using localfs
> (("path"="asterix_nc1://data/spatial/spatialData.json"),("format"="adm"));
> load dataset UpsertFrom
> using localfs
> (("path"="asterix_nc1://data/spatial/moreSpatialData.json"),("format"="adm"));
> upsert into dataset UpsertTo(
> for $x in dataset UpsertFrom
> return $x
> );
> for $o in dataset('UpsertTo')
> where spatial-intersect($o.point, 
> create-polygon([4.0,1.0,4.0,4.0,12.0,4.0,12.0,1.0]))
> order by $o.id
> return $o;
> {code}
> This DML returns the new record correctly. But, the issue is that the indexed 
> value in RTree has not been updated.
> When searching the rtree_index_point index, the searcher sees the previous 
> value - point("6.0,3.0"), not the new value - point("4.1,7.0"). So, this 
> record will be fetched from the primary index. However, the primary index 
> search returns the updated value and the select() verifies the value. It 
> returns true by coincidence because the new value satisfies the 
> spatial-intersect() condition.
> The secondary index search should see the updated value, not the previous 
> value. And this is an issue for the index-only plan case since it only uses 
> the value from a secondary index.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (ASTERIXDB-2176) Deletion doesn't work on the RTree index.

[Taewoo Kim (JIRA)]

[ 
https://issues.apache.org/jira/browse/ASTERIXDB-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265520#comment-16265520
 ] 

Taewoo Kim edited comment on ASTERIXDB-2176 at 11/24/17 8:35 PM:
-

Adding one more point:

An sqlpp execution test case: "scan-delete-rtree-secondary-index" conducts the 
following:

(1) creates a dataset 
(2) loads 21 records. (id = 1 ~ 21)
(3) creates a RTree index
(4) deletes 10 records (id > 10) 
(5) spatial-intersect query

On the master branch, an RTree search reveals id: 12 and 20. These records are 
filtered by the primary-index search since it cannot find the record 12 and 20. 
So it's not seen. Overall, I think deletion on RTree doesn't work now. And it 
is hidden because of the primary-index search.


was (Author: wangsaeu):
Adding one more point:

A sqlpp execution test case: "scan-delete-rtree-secondary-index" conducts the 
following:

(1) creates a dataset 
(2) loads 21 records. (id = 1 ~ 21)
(3) creates a RTree index
(4) deletes 10 records (id > 10) 
(5) query

On the master branch, an RTree search reveals id: 12 and 20. These records are 
filtered by the primary-index search since it cannot find the record 12 and 20. 
So it's not seen. Overall, I think deletion on RTree doesn't work now. And it 
is hidden because of the primary-index search.

> Deletion doesn't work on the RTree index.
> -
>
> Key: ASTERIXDB-2176
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2176
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Taewoo Kim
>
> This is a simplified version of "upsert/primary-secondary-tree" AQL test case.
> spatialData.json file
> { "id": 12, "point": point("6.0,3.0") }
> moreSpatialData.json file
> {"id": 12, "point": point("4.1,7.0")}
> DDL: 
> {code}
> drop dataverse test if exists;
> create dataverse test;
> use dataverse test;
> create type MyRecord as closed {
>   id: int64,
>   point: point
> }
> create dataset UpsertTo(MyRecord)
>  primary key id;
> create dataset UpsertFrom(MyRecord)
>  primary key id;
> create index rtree_index_point on UpsertTo(point) type rtree;
> {code}
> DML
> {code}
> load dataset UpsertTo
> using localfs
> (("path"="asterix_nc1://data/spatial/spatialData.json"),("format"="adm"));
> load dataset UpsertFrom
> using localfs
> (("path"="asterix_nc1://data/spatial/moreSpatialData.json"),("format"="adm"));
> upsert into dataset UpsertTo(
> for $x in dataset UpsertFrom
> return $x
> );
> for $o in dataset('UpsertTo')
> where spatial-intersect($o.point, 
> create-polygon([4.0,1.0,4.0,4.0,12.0,4.0,12.0,1.0]))
> order by $o.id
> return $o;
> {code}
> This DML returns the new record correctly. But, the issue is that the indexed 
> value in RTree has not been updated.
> When searching the rtree_index_point index, the searcher sees the previous 
> value - point("6.0,3.0"), not the new value - point("4.1,7.0"). So, this 
> record will be fetched from the primary index. However, the primary index 
> search returns the updated value and the select() verifies the value. It 
> returns true by coincidence because the new value satisfies the 
> spatial-intersect() condition.
> The secondary index search should see the updated value, not the previous 
> value. And this is an issue for the index-only plan case since it only uses 
> the value from a secondary index.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ASTERIXDB-2176) Deletion doesn't work on the RTree index.

[Taewoo Kim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ASTERIXDB-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taewoo Kim updated ASTERIXDB-2176:
--
Summary: Deletion doesn't work on the RTree index.  (was: Upsert doesn't 
work on the RTree index.)

> Deletion doesn't work on the RTree index.
> -
>
> Key: ASTERIXDB-2176
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-2176
> Project: Apache AsterixDB
>  Issue Type: Bug
>Reporter: Taewoo Kim
>
> This is a simplified version of "upsert/primary-secondary-tree" AQL test case.
> spatialData.json file
> { "id": 12, "point": point("6.0,3.0") }
> moreSpatialData.json file
> {"id": 12, "point": point("4.1,7.0")}
> DDL: 
> {code}
> drop dataverse test if exists;
> create dataverse test;
> use dataverse test;
> create type MyRecord as closed {
>   id: int64,
>   point: point
> }
> create dataset UpsertTo(MyRecord)
>  primary key id;
> create dataset UpsertFrom(MyRecord)
>  primary key id;
> create index rtree_index_point on UpsertTo(point) type rtree;
> {code}
> DML
> {code}
> load dataset UpsertTo
> using localfs
> (("path"="asterix_nc1://data/spatial/spatialData.json"),("format"="adm"));
> load dataset UpsertFrom
> using localfs
> (("path"="asterix_nc1://data/spatial/moreSpatialData.json"),("format"="adm"));
> upsert into dataset UpsertTo(
> for $x in dataset UpsertFrom
> return $x
> );
> for $o in dataset('UpsertTo')
> where spatial-intersect($o.point, 
> create-polygon([4.0,1.0,4.0,4.0,12.0,4.0,12.0,1.0]))
> order by $o.id
> return $o;
> {code}
> This DML returns the new record correctly. But, the issue is that the indexed 
> value in RTree has not been updated.
> When searching the rtree_index_point index, the searcher sees the previous 
> value - point("6.0,3.0"), not the new value - point("4.1,7.0"). So, this 
> record will be fetched from the primary index. However, the primary index 
> search returns the updated value and the select() verifies the value. It 
> returns true by coincidence because the new value satisfies the 
> spatial-intersect() condition.
> The secondary index search should see the updated value, not the previous 
> value. And this is an issue for the index-only plan case since it only uses 
> the value from a secondary index.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Change in asterixdb[master]: [NO ISSUE] Remove Algebricks examples

[Murtadha Hubail (Code Review)]

Murtadha Hubail has posted comments on this change.

Change subject: [NO ISSUE] Remove Algebricks examples
..


Patch Set 2: Code-Review+2

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2180
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I9465405d9e929161bcc6292c5f65c7f3f385c886
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Till Westmann 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Murtadha Hubail 
Gerrit-HasComments: No

Change in asterixdb[master]: [NO ISSUE] Remove Algebricks examples

[Anon. E. Moose (Code Review)]

Anon. E. Moose #1000171 has posted comments on this change.

Change subject: [NO ISSUE] Remove Algebricks examples
..


Patch Set 2: Contrib+2

Analytics Compatibility Tests Successful
https://goo.gl/A7rzQx: SUCCESS

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2180
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I9465405d9e929161bcc6292c5f65c7f3f385c886
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Till Westmann 
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins 
Gerrit-HasComments: No