[
https://issues.apache.org/jira/browse/PHOENIX-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665874#comment-16665874
]
ASF GitHub Bot commented on PHOENIX-4997:
-----------------------------------------
Github user karanmehta93 commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/397#discussion_r228699968
--- Diff:
phoenix-core/src/it/java/org/apache/phoenix/end2end/TableSnapshotReadsMapReduceIT.java
---
@@ -191,31 +192,52 @@ private void upsertData(PreparedStatement stmt,
String field1, String field2, in
stmt.execute();
}
- public void upsertAndSnapshot(String tableName) throws Exception {
+ private void upsertAndSnapshot(String tableName, boolean shouldSplit)
throws Exception {
upsertData(tableName);
+ TableName hbaseTableName = TableName.valueOf(tableName);
Connection conn = DriverManager.getConnection(getUrl());
Admin admin =
conn.unwrap(PhoenixConnection.class).getQueryServices().getAdmin();
- admin.snapshot(SNAPSHOT_NAME, TableName.valueOf(tableName));
- // call flush to create new files in the region
- admin.flush(TableName.valueOf(tableName));
+
+ if (shouldSplit) {
+ splitTableSync(admin, hbaseTableName, "BBBB".getBytes(), 2);
+ }
+
+ admin.snapshot(SNAPSHOT_NAME, hbaseTableName);
List<SnapshotDescription> snapshots = admin.listSnapshots();
Assert.assertEquals(tableName, snapshots.get(0).getTable());
+ // Capture the snapshot timestamp to use as SCN while reading the
table later
+ // Assigning the timestamp value here will make tests less flaky
+ timestamp = System.currentTimeMillis();
+
// upsert data after snapshot
PreparedStatement stmt = conn.prepareStatement(String.format(UPSERT,
tableName));
- upsertData(stmt, "DDDD", "SNFB", 0004);
+ upsertData(stmt, "DDDD", "SNFB", 45);
conn.commit();
}
- public void deleteSnapshot(String tableName) throws Exception {
+ private void splitTableSync(Admin admin, TableName hbaseTableName,
--- End diff --
Didn't use the method provided in BaseTest.java since enabling/disabling
the table causes the snapshot manifest file to not contain the offline regions.
The bug is caused because the snapshot manifest file on a regular
production table can contain information about regions that are split or
offline, which should ideally be ignored when restoring snapshot.
> Phoenix MR on snapshots can produce duplicate rows
> --------------------------------------------------
>
> Key: PHOENIX-4997
> URL: https://issues.apache.org/jira/browse/PHOENIX-4997
> Project: Phoenix
> Issue Type: Bug
> Reporter: Karan Mehta
> Assignee: Karan Mehta
> Priority: Major
>
> Phoenix MR over snapshots uses TableSnapshotResultIterator and
> SnapshotScanner classes for iterating/scanning over snapshots. They had been
> copied over from HBase classes TableSnapshotScanner and
> ClientSideRegionScanner classes and modified according to Phoenix
> requirements. This decision was taken since some of fields of these classes
> were private and hence it was not possible to reuse them. HBASE-8369 is the
> main Jira.
> The framework had a bug which was fixed as part of HBASE-16011. However the
> fix was not ported to Phoenix and hence Phoenix MR over snapshots still
> continues to have it. This Jira is to fix that issue.
> FYI [~akshita.malhotra]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
