[
https://issues.apache.org/jira/browse/APEXMALHAR-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15085437#comment-15085437
]
ASF GitHub Bot commented on APEXMALHAR-1963:
--------------------------------------------
Github user shubham-pathak22 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/157#discussion_r48948739
--- Diff:
library/src/main/java/com/datatorrent/lib/async/AbstractAsyncProcessor.java ---
@@ -0,0 +1,357 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package com.datatorrent.lib.async;
+
+import java.util.Iterator;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.Lists;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator.IdleTimeHandler;
+import com.datatorrent.api.annotation.InputPortFieldAnnotation;
+import com.datatorrent.common.util.BaseOperator;
+
+/**
+ * This base operator having a single input and tuple coming from input
port are enqueued for processing asynchronously
+ * by Thread Executors.
+ *
+ * User can configure number of of threads that are required for
processing. Defaulting to single thread which is
+ * different than operator thread.
+ * The user will also configure the timeout for the operations to happen.
+ *
+ * <b>Workflow is as follows:</b>
+ * <ol>
+ * <li>Tuple of type INPUT is received on input port.</li>
+ * <li>Tuple gets enqueued into Executors by calling
<i>enqueueTupleForProcessing</i>.
+ * Once can override <i>processTuple</i> method in which case, its
users responsibility to enqueue tuple for
+ * processing by calling <i>enqueueTupleForProcessing</i>.</li>
+ * <li>When tuple gets its turn of execution, <i>processTupleAsync</i>
gets called from thread other than
+ * operator thread.</li>
+ * <li>Based on maintainTupleOrder parameter, the results are
consolidated at end of every window and respective call
+ * to <i>handleProcessedTuple</i> will be made. This call will
happen in operator thread.</li>
+ * </ol>
+ *
+ * This operator can be implemented to do async and parallel operations.
+ *
+ * Use case examples:
+ * <ul>
+ * <li>Parallel reads from external datastore/database system.</li>
+ * <li>Doing any operations which are long running tasks per tuple but
DAG io should not be blocked.</li>
+ * </ul>
+ *
+ * @param <INPUT> input type of tuple
+ * @param <RESULT> Result of async processing of tuple.
+ * @since 3.3.0
+ */
+public abstract class AbstractAsyncProcessor<INPUT, RESULT> extends
BaseOperator implements IdleTimeHandler
+{
+ private static final Logger logger =
LoggerFactory.getLogger(AbstractAsyncProcessor.class);
+
+ private transient ExecutorService executor;
+ private int numProcessors = 1;
+ private boolean maintainTupleOrder = false;
+ private long processTimeoutMs = 0;
+
+ /**
+ * This Queue will hold all the tuples that are in process, has
completed processing but yet to be notified for
+ * processing completion.
+ */
+ private Queue<ProcessTuple> waitingTuples = Lists.newLinkedList();
+
+ @InputPortFieldAnnotation(optional = true)
--- End diff --
why optional ?
> Add abstract operator for Async Processing
> ------------------------------------------
>
> Key: APEXMALHAR-1963
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1963
> Project: Apache Apex Malhar
> Issue Type: New Feature
> Affects Versions: 3.3.0
> Reporter: Chinmay Kolhatkar
> Assignee: Chinmay Kolhatkar
>
> Create an abstract operator which does following:
> 1) Asynchronously processes the tuples
> 2) Have parallel executions
> 3) Make sure the outbound tuples are ordered similar to inbound tuples if
> configured.
> 4) Have processing timeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
