Can be done by sending "shutdown" message via heartbeat to Stram. Then on
stram can shutdown the entire app
Thks
Amol
On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda
wrote:
> Yes Ajay, for a graceful shutdown, the data sent out should be processed.
>
> On Wed, Jan 18, 2017 at 12:19 PM, AJAY G
[
https://issues.apache.org/jira/browse/APEXCORE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827550#comment-15827550
]
Amol Kekre commented on APEXCORE-503:
-
This is a good idea. We could have a way for
Yes Ajay, for a graceful shutdown, the data sent out should be processed.
On Wed, Jan 18, 2017 at 12:19 PM, AJAY GUPTA wrote:
> +1 to idea.
>
> Will this ensure downstream operators to process all data received before
> shutdown is called?
> Also, how do we plan to handle cases where 2 sub-DAGs
Hi Sanjay,
The task of the shutdown is to stop the processing from the input operator
itself. Once we reach a consistent state (endWindow() for example), we stop
the input operator. Thereafter, the next operators in the DAG would only
have to process the data which is already emitted by the input o
+1 to idea.
Will this ensure downstream operators to process all data received before
shutdown is called?
Also, how do we plan to handle cases where 2 sub-DAGs merge to a single
operator somewhere downstream, and an operator in one of the sub-DAGs sends
ShutdownException.
Ajay
On Wed, Jan 18, 2
If it is not an error condition, I would like for the data that is already
sent out by the upstream operators to get processed, before the shutdown of
an operator is initiated. Imagine something like a control tuple that
starts from the input operators and flows down the stream and stops the
contai
I have similar question/concerns. When a downstream operator initiates a
shutdown, how is that trigger signaled to and co-ordinated with the input
operators stopping their input reading activity? Or idempotency etc don’t
matter after shutdown?
On 1/17/17, 10:31 PM, "Chinmay Kolhatkar" wrote:
This JIRA is to stop the DAG in a crude manner, based on an error
condition. I think this might also need similar functionality as an error
condition can occur anywhere in the DAG.
Perhaps we can modify the same JIRA to include a graceful + ungraceful
(kill) shutdown from any operator in the DAG.
[
https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827493#comment-15827493
]
Ajay Gupta commented on APEXCORE-611:
-
Hi,
The below approach can be used for this
+1 for the feature in general.
Though I have a question about it: Lets say 10th operator in the DAG
initiates the shutdown.. And operator being in quite downstream as compared
to Input Operator, it might be fairly behind in window it is processing.
What happens to the data being already processed
I think this would be a great addition for batch use cases or use
cases were DAG needs to be shutdown after detecting some
completion/error condition through the operator. We have one Jira
Opened for such functionality
https://issues.apache.org/jira/browse/APEXCORE-503.
- Tushar.
On Wed, Jan 18,
Ajay Gupta created APEXCORE-611:
---
Summary: Stram Event Log Levels
Key: APEXCORE-611
URL: https://issues.apache.org/jira/browse/APEXCORE-611
Project: Apache Apex Core
Issue Type: Improvement
Hi All,
Currently we can shutdown an Apex app in the following ways:
1. Throw ShutdownException() from *all* the input operators
2. Use Apex CLI to shutdown an app using the YARN App Id
I think we should have some way of shutting down an application from within
an operator. It is not always true
[
https://issues.apache.org/jira/browse/APEXMALHAR-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827093#comment-15827093
]
ASF GitHub Bot commented on APEXMALHAR-2359:
GitHub user brightchen reope
[
https://issues.apache.org/jira/browse/APEXMALHAR-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827092#comment-15827092
]
ASF GitHub Bot commented on APEXMALHAR-2359:
Github user brightchen close
GitHub user brightchen reopened a pull request:
https://github.com/apache/apex-malhar/pull/518
APEXMALHAR-2359 #resolve #comment Optimise fire trigger to avoid go tâ¦
â¦hrough all data
You can merge this pull request into a Git repository by running:
$ git pull https://github
Github user brightchen closed the pull request at:
https://github.com/apache/apex-malhar/pull/518
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the featur
Yes, this can be part of operator configuration. Given this, for a user to
define a batch application, would mean configuring the connectors (mostly
the input operator) in the application for the desired behavior. Similarly,
there can be other use cases that can be achieved other than batch.
We ma
The HDFS source can operate in two modes, bounded or unbounded. If you scan
only once, then it should emit the final watermark after it is done.
Otherwise it would emit watermarks based on a policy (files names etc.).
The mechanism to generate the marks may depend on the type of source and
the user
+1
On Mon, Jan 16, 2017 at 1:23 AM, Chinmay Kolhatkar
wrote:
> Hi All,
>
> Currently a DAG that is generated by user, if contains any POJOfied
> operators, TUPLE_CLASS attribute needs to be set on each and every port
> which receives or sends a POJO.
>
> For e.g., if a DAG is like File -> Parser
GitHub user patilvikram opened a pull request:
https://github.com/apache/apex-malhar/pull/537
Fixed-APEXMALHAR-2389 Added user documentation for Apache Calcite
This will add user documentation for Apache Calcite usage in Apex
Applications.
You can merge this pull request into a Gi
[
https://issues.apache.org/jira/browse/APEXMALHAR-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826038#comment-15826038
]
ASF GitHub Bot commented on APEXMALHAR-2389:
GitHub user patilvikram open
Hi Thomas,
I am not sure that I completely understand your suggestion. Are you
suggesting to broaden the scope of the proposal to treat all sources as
bounded as well as unbounded?
In case of Apex, we treat all sources as unbounded sources. Even bounded
sources like HDFS file source is treated as
Github user patilvikram closed the pull request at:
https://github.com/apache/apex-malhar/pull/536
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the featu
+1
Regards,
Mohit
On Tue, Jan 17, 2017 at 12:11 PM, Bhupesh Chawda
wrote:
> +1 for the feature.
>
> ~ Bhupesh
>
> On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar
> wrote:
>
> > Those are not really anonymous POJOs... The definition of POJO will be
> > known to user as based on that only ups
vikram created APEXMALHAR-2389:
--
Summary: Add User Documentation for Calcite Integration
Key: APEXMALHAR-2389
URL: https://issues.apache.org/jira/browse/APEXMALHAR-2389
Project: Apache Apex Malhar
GitHub user patilvikram opened a pull request:
https://github.com/apache/apex-malhar/pull/536
User Documentation for Apache Calcite integration with Apache Apex
This request will cover detail description of Apache Calcite integration
with Apex. It will cover detail description about
Hi All,
Thank you for the opinions and suggestions. As per the consensus we will
continue to rely on the user to create dt_meta table.
Jira for documentation already exists (APEXMALHAR-2383) and is in progress.
Regards,
Hitesh
On Mon, Jan 16, 2017 at 9:16 PM, Thomas Weise wrote:
> For JDBC ex
[
https://issues.apache.org/jira/browse/APEXMALHAR-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825855#comment-15825855
]
ASF GitHub Bot commented on APEXMALHAR-2383:
GitHub user Hitesh-Scorpio o
GitHub user Hitesh-Scorpio opened a pull request:
https://github.com/apache/apex-malhar/pull/535
APEXMALHAR-2383 Documentation for Jdbc output operator
@amberarrow please review
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Hite
[
https://issues.apache.org/jira/browse/APEXMALHAR-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825758#comment-15825758
]
ASF GitHub Bot commented on APEXMALHAR-2354:
GitHub user bhupeshchawda op
GitHub user bhupeshchawda opened a pull request:
https://github.com/apache/apex-malhar/pull/534
APEXMALHAR-2354 Support for heuristic watermark
@davidyan74 @chinmaykolhatkar Please see.
You can merge this pull request into a Git repository by running:
$ git pull https://github.
[
https://issues.apache.org/jira/browse/APEXCORE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825658#comment-15825658
]
ASF GitHub Bot commented on APEXCORE-610:
-
GitHub user tushargosavi opened a pul
GitHub user tushargosavi opened a pull request:
https://github.com/apache/apex-core/pull/446
APEXCORE-610 Avoid multiple calls to getBytes.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tushargosavi/apex-core APEXCORE-610
Alte
Tushar Gosavi created APEXCORE-610:
--
Summary: Avoid multiple getBytes() calls in Tuple.writeString
Key: APEXCORE-610
URL: https://issues.apache.org/jira/browse/APEXCORE-610
Project: Apache Apex Core
35 matches
Mail list logo