Github user vipul1409 commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2922#discussion_r90598402
  
    --- Diff: 
flink-contrib/flink-streaming-contrib/src/main/java/org/apache/flink/contrib/streaming/MultiThreadedFlatMapFunction.java
 ---
    @@ -0,0 +1,91 @@
    +package org.apache.flink.contrib.streaming;
    +
    +import org.apache.flink.api.common.functions.FlatMapFunction;
    +import org.apache.flink.api.common.functions.RichFlatMapFunction;
    +import org.apache.flink.api.common.functions.util.ListCollector;
    +import org.apache.flink.configuration.Configuration;
    +import org.apache.flink.runtime.util.NamedThreadFactory;
    +import org.apache.flink.util.Collector;
    +
    +import java.util.ArrayList;
    +import java.util.List;
    +import java.util.concurrent.*;
    +
    +/**
    + * Created by vipulmodi on 11/30/16.
    + */
    +
    +//To return Futures to calling fucntion we can do OUT extends Future.
    +public class MultiThreadedFlatMapFunction<IN, OUT> extends 
RichFlatMapFunction<IN, OUT> {
    +
    +    private static final long serialVersionUID;
    +    private ExecutorService flatMapExecutors;
    +    private Integer numberOfthreads;
    +    private List<Callable<ArrayList<OUT>>> callables;
    +    private List<Future<ArrayList<OUT>>> futures;
    +    static {
    +        serialVersionUID = 1L;
    +    }
    +
    +    public FlatMapFunction flatMapFunction;
    +
    +    public MultiThreadedFlatMapFunction(FlatMapFunction flatMapFunction, 
Integer numberOfthreads) {
    +        this.flatMapFunction = flatMapFunction;
    +        this.numberOfthreads = numberOfthreads;
    +        this.callables = new ArrayList<>();
    +        this.futures = new ArrayList<>();
    +    }
    +
    +    // Understand side effects of making out final.
    +    @Override
    +    public void flatMap(final IN value, final Collector<OUT> out) throws 
Exception {
    +        // Figure out a better way to call
    +        /*callables.add(new Callable<ArrayList<OUT>>() {
    +            @Override
    +            public ArrayList<OUT> call() throws Exception {
    +                ArrayList<OUT> result = new ArrayList<OUT>();
    +                Collector<OUT> collector = new ListCollector<OUT>(result);
    +                flatMapFunction.flatMap(value, collector);
    +                return result;
    +            }
    +        });*/
    +
    +        futures.add(flatMapExecutors.submit(new Callable<ArrayList<OUT>>() 
{
    +            @Override
    +            public ArrayList<OUT> call() throws Exception {
    +                ArrayList<OUT> result = new ArrayList<OUT>();
    +                Collector<OUT> collector = new ListCollector<OUT>(result);
    --- End diff --
    
    Any suggestions on which collector would be a good choice over here. I am 
not aware of all the options. Is there any document available for different 
collectors.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Reply via email to