[
https://issues.apache.org/jira/browse/APEXMALHAR-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306226#comment-15306226
]
ASF GitHub Bot commented on APEXMALHAR-2103:
--------------------------------------------
Github user chaithu14 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/300#discussion_r65023539
--- Diff:
library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java ---
@@ -375,11 +374,11 @@ public void run()
lastScannedInfo = null;
numDiscoveredPerIteration = 0;
for (String afile : files) {
- String filePath = new File(afile).getAbsolutePath();
- LOG.debug("Scan started for input {}", filePath);
+ Path filePath = new Path(afile);
+ LOG.debug("Scan started for input {}", filePath.toString());
Map<String, Long> lastModifiedTimesForInputDir;
- lastModifiedTimesForInputDir = referenceTimes.get(filePath);
- scan(new Path(afile), null, lastModifiedTimesForInputDir);
+ lastModifiedTimesForInputDir =
referenceTimes.get(fs.getFileStatus(filePath).getPath().toString());
--- End diff --
No. In case of LocalFileSystem, if the filePath consists of relative path
then again the keys are different.
Setting the directory path in createScannedFileInfo() and accessing the
value from referenceTimes must be sync.
> scanner issues in FileSplitterInput class
> -----------------------------------------
>
> Key: APEXMALHAR-2103
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2103
> Project: Apache Apex Malhar
> Issue Type: Bug
> Reporter: Chaitanya
> Assignee: Chaitanya
>
> Issue: FileSplitter continuously emitting filemetadata even though there is
> a single file.
> Observation: For the same file, While updating and accessing the
> referenceTimes map in FIleSplitterInput and TimeBasedScanner, the Keys are
> different. Because of this, the oldestTimeModification is always null in
> TimeBasedScanner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
