Re: Shutdown of an Apex app

I think this would be a great addition for batch use cases or use cases were DAG needs to be shutdown after detecting some completion/error condition through the operator. We have one Jira Opened for such functionality https://issues.apache.org/jira/browse/APEXCORE-503. - Tushar. On Wed, Jan

Re: Shutdown of an Apex app

I have similar question/concerns. When a downstream operator initiates a shutdown, how is that trigger signaled to and co-ordinated with the input operators stopping their input reading activity? Or idempotency etc don’t matter after shutdown? On 1/17/17, 10:31 PM, "Chinmay Kolhatkar"

Re: Shutdown of an Apex app

Hi Sanjay, The task of the shutdown is to stop the processing from the input operator itself. Once we reach a consistent state (endWindow() for example), we stop the input operator. Thereafter, the next operators in the DAG would only have to process the data which is already emitted by the input

[jira] [Created] (APEXCORE-611) Stram Event Log Levels

Ajay Gupta created APEXCORE-611: --- Summary: Stram Event Log Levels Key: APEXCORE-611 URL: https://issues.apache.org/jira/browse/APEXCORE-611 Project: Apache Apex Core Issue Type: Improvement

Re: Shutdown of an Apex app

Yes Ajay, for a graceful shutdown, the data sent out should be processed. On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA wrote: > +1 to idea. > > Will this ensure downstream operators to process all data received before > shutdown is called? > Also, how do we plan to handle

[jira] [Commented] (APEXCORE-503) support KillException

[ https://issues.apache.org/jira/browse/APEXCORE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827550#comment-15827550 ] Amol Kekre commented on APEXCORE-503: - This is a good idea. We could have a way for operator to

Shutdown of an Apex app

Hi All, Currently we can shutdown an Apex app in the following ways: 1. Throw ShutdownException() from *all* the input operators 2. Use Apex CLI to shutdown an app using the YARN App Id I think we should have some way of shutting down an application from within an operator. It is not always true

[jira] [Commented] (APEXCORE-611) Stram Event Log Levels

[ https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827493#comment-15827493 ] Ajay Gupta commented on APEXCORE-611: - Hi, The below approach can be used for this JIRA. Kindly let

Re: Shutdown of an Apex app

This JIRA is to stop the DAG in a crude manner, based on an error condition. I think this might also need similar functionality as an error condition can occur anywhere in the DAG. Perhaps we can modify the same JIRA to include a graceful + ungraceful (kill) shutdown from any operator in the DAG.

Re: Shutdown of an Apex app

+1 for the feature in general. Though I have a question about it: Lets say 10th operator in the DAG initiates the shutdown.. And operator being in quite downstream as compared to Input Operator, it might be fairly behind in window it is processing. What happens to the data being already processed

Re: Shutdown of an Apex app

If it is not an error condition, I would like for the data that is already sent out by the upstream operators to get processed, before the shutdown of an operator is initiated. Imagine something like a control tuple that starts from the input operators and flows down the stream and stops the

Re: Schema Discovery Support in Apex Applications

+1 On Mon, Jan 16, 2017 at 1:23 AM, Chinmay Kolhatkar wrote: > Hi All, > > Currently a DAG that is generated by user, if contains any POJOfied > operators, TUPLE_CLASS attribute needs to be set on each and every port > which receives or sends a POJO. > > For e.g., if a DAG

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

The HDFS source can operate in two modes, bounded or unbounded. If you scan only once, then it should emit the final watermark after it is done. Otherwise it would emit watermarks based on a policy (files names etc.). The mechanism to generate the marks may depend on the type of source and the

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

Yes, this can be part of operator configuration. Given this, for a user to define a batch application, would mean configuring the connectors (mostly the input operator) in the application for the desired behavior. Similarly, there can be other use cases that can be achieved other than batch. We

[GitHub] apex-malhar pull request #518: APEXMALHAR-2359 #resolve #comment Optimise fi...

GitHub user brightchen reopened a pull request: https://github.com/apache/apex-malhar/pull/518 APEXMALHAR-2359 #resolve #comment Optimise fire trigger to avoid go t… …hrough all data You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (APEXMALHAR-2359) Optimise fire trigger to avoid go through all data

[ https://issues.apache.org/jira/browse/APEXMALHAR-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827092#comment-15827092 ] ASF GitHub Bot commented on APEXMALHAR-2359: Github user brightchen closed the pull

[jira] [Commented] (APEXMALHAR-2359) Optimise fire trigger to avoid go through all data

[ https://issues.apache.org/jira/browse/APEXMALHAR-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827093#comment-15827093 ] ASF GitHub Bot commented on APEXMALHAR-2359: GitHub user brightchen reopened a pull

[GitHub] apex-malhar pull request #518: APEXMALHAR-2359 #resolve #comment Optimise fi...

Github user brightchen closed the pull request at: https://github.com/apache/apex-malhar/pull/518 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[jira] [Created] (APEXCORE-610) Avoid multiple getBytes() calls in Tuple.writeString

Tushar Gosavi created APEXCORE-610: -- Summary: Avoid multiple getBytes() calls in Tuple.writeString Key: APEXCORE-610 URL: https://issues.apache.org/jira/browse/APEXCORE-610 Project: Apache Apex Core

[jira] [Commented] (APEXCORE-610) Avoid multiple getBytes() calls in Tuple.writeString

[ https://issues.apache.org/jira/browse/APEXCORE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825658#comment-15825658 ] ASF GitHub Bot commented on APEXCORE-610: - GitHub user tushargosavi opened a pull request:

[GitHub] apex-core pull request #446: APEXCORE-610 Avoid multiple calls to getBytes.

GitHub user tushargosavi opened a pull request: https://github.com/apache/apex-core/pull/446 APEXCORE-610 Avoid multiple calls to getBytes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tushargosavi/apex-core APEXCORE-610

[jira] [Commented] (APEXMALHAR-2354) Add support for heuristic watermarks in WindowedOperator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825758#comment-15825758 ] ASF GitHub Bot commented on APEXMALHAR-2354: GitHub user bhupeshchawda opened a pull

[jira] [Commented] (APEXMALHAR-2389) Add User Documentation for Calcite Integration

[ https://issues.apache.org/jira/browse/APEXMALHAR-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826038#comment-15826038 ] ASF GitHub Bot commented on APEXMALHAR-2389: GitHub user patilvikram opened a pull

Re: Schema Discovery Support in Apex Applications

+1 Regards, Mohit On Tue, Jan 17, 2017 at 12:11 PM, Bhupesh Chawda wrote: > +1 for the feature. > > ~ Bhupesh > > On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar > wrote: > > > Those are not really anonymous POJOs... The definition of POJO will

[GitHub] apex-malhar pull request #537: Fixed-APEXMALHAR-2389 Added user documentatio...

GitHub user patilvikram opened a pull request: https://github.com/apache/apex-malhar/pull/537 Fixed-APEXMALHAR-2389 Added user documentation for Apache Calcite This will add user documentation for Apache Calcite usage in Apex Applications. You can merge this pull request into a

[jira] [Created] (APEXMALHAR-2389) Add User Documentation for Calcite Integration

vikram created APEXMALHAR-2389: -- Summary: Add User Documentation for Calcite Integration Key: APEXMALHAR-2389 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2389 Project: Apache Apex Malhar

[GitHub] apex-malhar pull request #536: User Documentation for Apache Calcite integra...

Github user patilvikram closed the pull request at: https://github.com/apache/apex-malhar/pull/536 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

Hi Thomas, I am not sure that I completely understand your suggestion. Are you suggesting to broaden the scope of the proposal to treat all sources as bounded as well as unbounded? In case of Apex, we treat all sources as unbounded sources. Even bounded sources like HDFS file source is treated

[GitHub] apex-malhar pull request #535: APEXMALHAR-2383 Documentation for Jdbc output...

GitHub user Hitesh-Scorpio opened a pull request: https://github.com/apache/apex-malhar/pull/535 APEXMALHAR-2383 Documentation for Jdbc output operator @amberarrow please review You can merge this pull request into a Git repository by running: $ git pull

[jira] [Commented] (APEXMALHAR-2383) Add documentation for JdbcPOJOInsertOutputOperator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825855#comment-15825855 ] ASF GitHub Bot commented on APEXMALHAR-2383: GitHub user Hitesh-Scorpio opened a pull

Re: APEXMALHAR-2382 User needs to create dt_meta table while using JdbcPOJOInsertOutputOperator

Hi All, Thank you for the opinions and suggestions. As per the consensus we will continue to rely on the user to create dt_meta table. Jira for documentation already exists (APEXMALHAR-2383) and is in progress. Regards, Hitesh On Mon, Jan 16, 2017 at 9:16 PM, Thomas Weise

[GitHub] apex-malhar pull request #536: User Documentation for Apache Calcite integra...

GitHub user patilvikram opened a pull request: https://github.com/apache/apex-malhar/pull/536 User Documentation for Apache Calcite integration with Apache Apex This request will cover detail description of Apache Calcite integration with Apex. It will cover detail description

[GitHub] apex-malhar pull request #534: APEXMALHAR-2354 Support for heuristic waterma...

GitHub user bhupeshchawda opened a pull request: https://github.com/apache/apex-malhar/pull/534 APEXMALHAR-2354 Support for heuristic watermark @davidyan74 @chinmaykolhatkar Please see. You can merge this pull request into a Git repository by running: $ git pull