[
https://issues.apache.org/jira/browse/FLUME-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johny Rufus updated FLUME-2346:
-------------------------------
Attachment: FLUME-2346.patch
> idLogFileMap in Log can lose track of file ids
> ----------------------------------------------
>
> Key: FLUME-2346
> URL: https://issues.apache.org/jira/browse/FLUME-2346
> Project: Flume
> Issue Type: Bug
> Components: File Channel
> Affects Versions: v1.4.0
> Reporter: Steve Zesch
> Attachments: FLUME-2346.patch
>
>
> The following code from Log's writeCheckpoint method can lose track of file
> ids to Reader mappings which will lead to a NPE being thrown in subsequent
> calls to writeCheckpoint.
> {code:title=Log#writeCheckpoint}
> Iterator<Integer> idIterator = logFileRefCountsAll.iterator();
> while (idIterator.hasNext()) {
> int id = idIterator.next();
> LogFile.RandomReader reader = idLogFileMap.remove(id);
> File file = reader.getFile();
> reader.close();
> LogFile.MetaDataWriter writer =
> LogFileFactory.getMetaDataWriter(file, id);
> try {
> writer.markCheckpoint(logWriteOrderID);
> } finally {
> writer.close();
> }
> reader = LogFileFactory.getRandomReader(file, encryptionKeyProvider);
> idLogFileMap.put(id, reader);
> LOGGER.debug("Updated checkpoint for file: " + file
> + "logWriteOrderID " + logWriteOrderID);
> idIterator.remove();
> }
> {code}
> The problem occurs when writer.markCheckpoint throws an exception and the id
> -> reader mapping is not added back to idLogFileMap. The next time
> writeCheckpoint is called logFileRefCountsAll still contains the file id, but
> idLogFileMap.remove(id) returns null since the id is no longer in the map.
> Attempting to call reader.getFile() then throws a NPE.
> Is there a reason that the initial reader obtained from idLogFileMap is
> closed and then a new reader for the same file is later created an inserted
> into the map? If that is not required, then one possible solution would be to
> remove this logic and not remove the id -> reader mapping in idLogFileMap. If
> that logic is required, then perhaps the code to insert a new id -> reader
> mapping into idLogFileMap could be added to the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
