[
https://issues.apache.org/jira/browse/STORM-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated STORM-2355:
----------------------------------
Labels: pull-request-available (was: )
> Storm-HDFS: inotify support
> ---------------------------
>
> Key: STORM-2355
> URL: https://issues.apache.org/jira/browse/STORM-2355
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-hdfs
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This is a proposal to implement inotify based watch dir monitoring in
> Storm-HDFS Spout.
> *Motivation*
> Storm-HDFS's HdfsSpout currently polls the Spout’s input directory using
> Hadoop's {{FileSystem.listFiles()}}. This operation is expensive since it
> returns the block locations and all stat information of the files inside the
> watch directory. Moreover HdfsSpout currently uses only one element of the
> returned Path list which is inefficient as the rest of the entries are thrown
> away without processing.
> The proposed design provides greater efficiency through the inotify interface
> and also enables to easier extension of the original ({{listFiles()}} based)
> monitoring with buffering (see Further work section below).
> *High level design*
> Goal is to leverage [HDFS inotify
> API|http://hadoop.apache.org/docs/current/api//org/apache/hadoop/hdfs/DFSInotifyEventInputStream.html]
> to monitor new file arrival to HdfsSpout's input directory.
> The inotify based monitoring is an addition to the original
> {{FileSystem.listFiles()}} based implementation, the default behavior of the
> spout will be unchanged by this modification.
> To unify the two monitoring methods and enable buffering an iterator based
> ({{HdfsDirectoryMonitor}}) class is created.
> To retain backward compatibility the HdfsSpout's default monitoring behavior
> is unchanged, inotify based monitoring could be enabled through a parameter.
> As inotify requires administrative privileges (see Caveat section below) a
> fallback mechanism is be implemented in HdfsSpout to use the original
> {{listFiles()}} based monitoring if initialization fails for inotify based
> monitoring.
> *Implementation details*
> As inotify provides only a delta of the filesystem events from a given Tx Id
> (of Hdfs Edit Log) it is required to do a {{FileSystem.listFiles()}} based
> collection during the Spout's initialization to ensure that any left over
> files are processed.
> The inotify based implementation uses HdfsAdmin's
> [{{DFSInotifyEventInputStream.poll()}}|http://hadoop.apache.org/docs/current/api//org/apache/hadoop/hdfs/DFSInotifyEventInputStream.html#poll--]
> method to fetch and buffer the list of new files created since the provided
> Tx Id to {{newFileList}} buffer.
> During {{HdfsSpout.nextTuple()}} call one element is taken from the
> {{newFileList}} buffer and processed by the spout.
> The {{newFileList}} buffer is extended with the result of the
> {{DFSInotifyEventInputStream.poll(lastTxId)}} call in every nextTuple() call.
> Since HdfsSpout is able to create it's own {{HdfsAdmin()}} instance there
> will be no need for the user to do additional initialization for the spout
> even if inotify is enabled.
> *Caveat*
> HDFS inotify is currently available through hdfs administrator user only, but
> there is ongoing discussion in Hadoop community to extend its support to
> users. See: HDFS-8940
> *Further work*
> 1) The number of calls to {{DFSInotifyEventInputStream.poll(lastTxId)}} could
> be further reduced if the locking directory is moved away from the input
> directory. With the current design updates on the lock dir are also included
> in the {{newFileList}} buffer hence the buffer will never get completely
> empty.
> 2) The original {{listFiles()}} based solution could be improved through
> {{HdfsDirectoryMonitor}} to buffer and use all the returned items from the
> work directory, similarly to inotify based monitoring. Such improvement will
> reduce the number of calls made to namenode.
> These improvements are currently not part of this ticket.
> *Error scenarios*
> - Inability of HdfsAdmin instance creation (e.g. lack of privileges):
> The spout falls back to the original {{listFiles()}} based method.
> - Namenode's edit log is not yet open for write during {{HdfsSpout.open()}}:
> The initialization will be postponed to the {{HdfsSpout.nextTuple()}}
> call(s).
> - Hdfs gets disconnected while the topology is running:
> HdfsSpout reports an error and retries in the next call of nextSpout()
> call.
> No data will be skipped as the update will be requested from the last
> known Tx Id.
>
> *Testing related changes*
> The {{TestHdfsSpout}} testcase should be parametrized to check for both the
> poll & inotify based solution.
> Additional testcases are added to ensure that inotify is able to pick up any
> leftover files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
