-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6411/
-----------------------------------------------------------
(Updated Aug. 12, 2012, 5:54 p.m.)
Review request for Flume.
Changes
-------
Sorry to update again...I realized that the changes increased the likelihood
that simply using current time in ms as an increasing identifier would become
an issue and as such we now ensure that the identifiers txid/writer order id
are higher than anything found in the log files on replay.
Description
-------
This resolves FLUME-1432 which describes a problem with how we replayed logs.
Instead of replaying one log at a time, we replay all the logs in the order
they were written. We do this by changing the "timestamp" field to be a
globally ordered number which allows us to do a merge of all the log files.
This should be faster and also fix some problems where we could not replay some
checkpoints.
Its worth noting that the condition in testRaceFoundInFLUME1432 was observed
prior to this patch and will not pass if we simply used the transaction id to
order edits due to that being generated on transaction create or the previous
value of the field, timestamp, because they were observed to be non-unique on a
2 core host.
If committed FLUME-1433 can be closed as well since this change fixes that test
as well.
This addresses bug FLUME-1432.
https://issues.apache.org/jira/browse/FLUME-1432
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java
03b1060
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
cc8f89a
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java
3d3b9b3
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
e692934
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
2b733b1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
5615c6d
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogRecord.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
bcd37ab
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
da2d068
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java
b42501f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java
42b197f
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
c222bd1
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionIDOracle.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/WriteOrderOracle.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
17a7cf9
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
681ebcc
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
b243f5a
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
8995089
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogRecord.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
a46526d
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionIDOracle.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestWriteOrderOracle.java
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-checkpoint.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-log-1.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-log-2.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-log-3.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-checkpoint.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-log-1.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-log-2.gz
PRE-CREATION
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-log-3.gz
PRE-CREATION
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
641952a
Diff: https://reviews.apache.org/r/6411/diff/
Testing
-------
All unit tests pass. A unit test is added, testRaceFoundInFLUME1432 which found
a problem with the way we previously replayed logs. It passes with these
changes. I also tested on a two node flume installation stopping and restarting
a node several times.
Thanks,
Brock Noland
