Repository: apex-malhar Updated Branches: refs/heads/master ea1b58f72 -> c4f788d5a
APEXMALHAR-2312 Fix NullPointerException for FileSplitterInput Operator if filepath is specified. Problem Description: ------------------- 1) TimeBasedDirectoryScanner threads part of scanservice tries to scan the directories/files. 2) Each thread checks with help of isIterationCompleted() [referenceTimes] method whether scanned of last iteration are processed by operator thread. 3) Previously it used to work because HashMap (referenceTimes) used to return null even if last scanned directory path is null. 4) Recently referenceTimes is changed to ConcurrentHashMap, so get() doesn't allow null key's passed to ConcurrentHashMap get() method. 5) Hence NullPointerException is seen as if only file path is provided directory path would be empty hence key would be empty. Solution: --------- Pre-check that directory path is null then we have completed last iterations if only filepath is provided. Project: http://git-wip-us.apache.org/repos/asf/apex-malhar/repo Commit: http://git-wip-us.apache.org/repos/asf/apex-malhar/commit/c4f788d5 Tree: http://git-wip-us.apache.org/repos/asf/apex-malhar/tree/c4f788d5 Diff: http://git-wip-us.apache.org/repos/asf/apex-malhar/diff/c4f788d5 Branch: refs/heads/master Commit: c4f788d5a140bcc14bf9782f4021c0af4712f075 Parents: ea1b58f Author: deepak-narkhede <[email protected]> Authored: Fri Oct 21 12:14:34 2016 +0530 Committer: deepak-narkhede <[email protected]> Committed: Thu Oct 27 16:02:32 2016 +0530 ---------------------------------------------------------------------- .../com/datatorrent/lib/io/fs/FileSplitterInput.java | 12 ++++++++++++ 1 file changed, 12 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/apex-malhar/blob/c4f788d5/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java ---------------------------------------------------------------------- diff --git a/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java b/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java index 745f953..985c667 100644 --- a/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java +++ b/library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java @@ -393,6 +393,18 @@ public class FileSplitterInput extends AbstractFileSplitter implements InputOper if (lastScannedInfo == null) { // first iteration started return true; } + + LOG.debug("Directory path: {} Sub-Directory or File path: {}", lastScannedInfo.getDirectoryPath(), lastScannedInfo.getFilePath()); + + /* + * As referenceTimes is now concurrentHashMap, it throws exception if key passed is null. + * So in case where the last scanned directory is null which likely possible when + * only file name is specified instead of directory path. + */ + if (lastScannedInfo.getDirectoryPath() == null) { + return true; + } + Map<String, Long> referenceTime = referenceTimes.get(lastScannedInfo.getDirectoryPath()); if (referenceTime != null) { return referenceTime.get(lastScannedInfo.getFilePath()) != null;
