[
https://issues.apache.org/jira/browse/APEXMALHAR-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290345#comment-15290345
]
ASF GitHub Bot commented on APEXMALHAR-2096:
--------------------------------------------
Github user gauravgopi123 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/283#discussion_r63816258
--- Diff:
library/src/main/java/com/datatorrent/lib/io/fs/FSInputModule.java ---
@@ -107,6 +110,9 @@ public void populateDAG(DAG dag, Configuration conf)
dag.setAttribute(blockReader, Context.OperatorContext.PARTITIONER,
new StatelessPartitioner<FSSliceReader>(readersCount));
fileSplitter.setBlocksThreshold(readersCount);
}
+ if (blocksThreshold != 0) {
--- End diff --
Do you need this check as blocksThreshold is always >=1 according to
constraint?
> Add blockThreshold parameter to FSInputModule
> ---------------------------------------------
>
> Key: APEXMALHAR-2096
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2096
> Project: Apache Apex Malhar
> Issue Type: Improvement
> Reporter: Priyanka Gugale
> Assignee: Priyanka Gugale
>
> FileSplitter is very fast, the downstream operators can't work at that speed,
> so we need to limit the speed with which fileSplitter works.
> One way to do is limit number of blocks emitted per window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
