[
https://issues.apache.org/jira/browse/BLUR-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739336#comment-13739336
]
Rahul Challapalli commented on BLUR-132:
----------------------------------------
Thanks for the reply Aaron. I understood the task differently(to take backups).
Thanks for the clarification. Below are few of my thoughts.
We might not need to add/modify any IndexDeletionPolicy if my understanding is
correct
We can use the SnapshotDeletionPolicy(wrapping
KeepOnlyLastCommittedIndexDeletionPolicy) to snapshot the last commit.
SnapshotDeletionPolicy controls what IndexCommit's are passed to
KeepOnlyLastCommittedIndexDeletionPolicy and does not include the
snapshotted IndexCommits. If we do not call the release method then the
snapshotted index commit will not be deleted
This is how we instantiate the SnapshotDeletionPolicy (so that snapshotted
IndexCommits do not get deleted upon shardserver restart)
SnapshotDeletionPolicy snapshotter = new SnapshotDeletionPolicy(new
KeepOnlyLastCommittedIndexDeletionPolicy(), snapshotsInfo);
Below is some pseudo code
DistributedIndexServer {
Map<String, String> snapshotsInfo; // snapshotId --> segmentsFilename
public void init() {
...
...
loadSnapshotsInfo();
}
public String snapshot() {
try {
for(String table : _clusterStatus.getTableList(true)) {
Map<String, BlurIndex> indexes = getIndexes(table);
for (String shard : indexes.keySet()) {
BlurIndex index = indexes.get(shard);
// Not sure how to get the writer.....BlurIndexReader can return null
may be?
BlurIndexWriter writer = index.getWriter();
if (writer != null) {
writer.commit();
SnapshotDeletionPolicy snapshotter =
(SnapshotDeletionPolicy)writer.getConfig().getIndexDeletionPolicy();
String commitIdentifier = generateCommitIdentifier();
IndexCommit indexCommit = snapshotter.snapshot(commitIdentifier);
_snapshotsInfo.put(commitIdentifier,
indexCommit.getSegmentsFileName());
// not sure where to persistently store _snapshotsInfo to load it
on shard server restart
return commitIdentifier;
} else {
// index is being updated....we have an instance of BlurIndexReader
LOG.error("Cannot take a snapshot as the Index is being updated on
" + table + "/" + shard);
throw new IOException("Cannot take a snapshot as the Index is being
updated on " + table + "/" + shard);
}
}
}
} catch (Exception e) {
LOG.error("Unknown error while trying to get a table list.", e);
throw new IOException("Unknown error while trying to get a table list.",
e);
}
}
private String generateCommitIdentifier() {
return UUID.randomUUID().toString();
}
}
If we want to read from a particular snapshot we can use
DirectoryReader.open(indexCommit) in BlurIndexReader.
Sorry for the huge comment and random pseudo code. I don't want to get too far
if I am missing something
- Rahul
> Create Index Snapshots
> ----------------------
>
> Key: BLUR-132
> URL: https://issues.apache.org/jira/browse/BLUR-132
> Project: Apache Blur
> Issue Type: New Feature
> Affects Versions: 0.3.0
> Reporter: Aaron McCurry
> Fix For: 0.3.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
