Re: Core release 3.7.0

+1 for release ~ Bhupesh On Jan 19, 2018 12:14 AM, "Ananth G" wrote: > +1 for the release. > > Regards, > Ananth > > On Fri, Jan 19, 2018 at 4:41 AM, Chinmay Kolhatkar > wrote: > > > +1 for the release. > > > > - Chinmay. > > > > On 18 Jan 2018 8:58

Re: [VOTE] Apache Apex Malhar 3.8.0 release candidate RC1

+1 - Signatures verified - Build successful without tests - LICENSE, NOTICE and README.md exists ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Thu, Nov 9, 2017

Re: [ANNOUNCE] New Apache Apex PMC: Tushar Gosavi

Congratulations Tushar! ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Fri, Nov 3, 2017 at 9:54 PM, Hitesh Kapoor <hit...@datatorrent.com>

Re: [ANNOUNCE] New Apache Apex Committer: Ananth Gundabattula

Congratulations Ananth! ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Fri, Nov 3, 2017 at 9:37 PM, Tushar Gosavi <tus...@datatorrent.com>

Re: Malhar release 3.8.0

+1 for malhar release ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Wed, Oct 25, 2017 at 9:37 PM, Chinmay Kolhatkar <chin...@datatorrent.com> wrote:

Re: Difference between setup() and activate()

There's one more reason why you might need to initialize your transients in activate rather than setup. If you are working with pojos and depend on the class of the a tuple via a configured port attribute (TUPLE_CLASS), then this class will only be available in the activate method, and not in the

Re: Input needed for ApexCli.

+1 for best effort with warnings. ~ Bhupesh On Aug 4, 2017 23:46, "Pramod Immaneni" wrote: > I would prefer "Best effort" with warnings for the ones that are invalid. > > On Fri, Aug 4, 2017 at 9:42 AM, Florian Schmidt > wrote: > > > Hey

[jira] [Updated] (APEXMALHAR-2532) Transform Application Test flooding CI logs

[ https://issues.apache.org/jira/browse/APEXMALHAR-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXMALHAR-2532: --- Labels: newbie (was: ) > Transform Application Test flooding CI l

[jira] [Resolved] (APEXCORE-602) Provide a "group-id" in the event object so that events are grouped together by a "root cause".

[ https://issues.apache.org/jira/browse/APEXCORE-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXCORE-602. - Resolution: Fixed Fix Version/s: 3.7.0 > Provide a "group-id"

[jira] [Commented] (APEXMALHAR-2507) Example for inner join functionality using Windowed merge operator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053535#comment-16053535 ] Bhupesh Chawda commented on APEXMALHAR-2507: I agree. The current example need

[jira] [Resolved] (APEXMALHAR-2366) Apply BloomFilter to Bucket

[ https://issues.apache.org/jira/browse/APEXMALHAR-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXMALHAR-2366. Resolution: Done Fix Version/s: 3.8.0 > Apply BloomFilter to Buc

[jira] [Created] (APEXMALHAR-2507) Example for inner join functionality using Windowed merge operator

Bhupesh Chawda created APEXMALHAR-2507: -- Summary: Example for inner join functionality using Windowed merge operator Key: APEXMALHAR-2507 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2507

Re: File control tuples

he.org> wrote: The extra port seems unnecessary unless you are planning to associate each individual data tuple with a file reference (similar to WindowedTuple)? On Sat, Jun 3, 2017 at 8:50 PM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > This is not specific to the batch wor

Re: File control tuples

This is not specific to the batch work. This is a more generic functionality which even streaming applications can benefit from. The separate port is for both the actual tuple as well as the metadata. ~ Bhupesh ___ Bhupesh Chawda E: bhup

File control tuples

with the actual tuple. We can try to reduce the amount of meta data that goes with each tuple by having some sort of meta encoding in the control tuple. ~ Bhupesh​ ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc

Re: license issue for APEXMALHAR-2366

___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Thu, May 25, 2017 at 3:45 PM, Pramod Immaneni <pra...@datatorrent.com> wrote: > Hi, > > The PR 521 <https://github.com/apache/apex-malhar/pull/521> for >

Re: Backward compatibility issue in 3.6.0 release

___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Mon, May 15, 2017 at 11:40 AM, Vlad Rozov <v.ro...@datatorrent.com> wrote: > I am slightly in favor of getters and setters approach as well. Likely > hotspot will optimiz

[jira] [Commented] (APEXMALHAR-2503) CI Flake: org.apache.apex.malhar.lib.join.POJOPartitionJoinOperatorTest

[ https://issues.apache.org/jira/browse/APEXMALHAR-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028012#comment-16028012 ] Bhupesh Chawda commented on APEXMALHAR-2503: Please see my comment on https

[jira] [Commented] (APEXMALHAR-2488) Simplify join support in Malhar

[ https://issues.apache.org/jira/browse/APEXMALHAR-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16028010#comment-16028010 ] Bhupesh Chawda commented on APEXMALHAR-2488: I think we need to deprecate the inner join

Re: Backward compatibility issue in 3.6.0 release

I also suggest going for 1. But I think we should make variables private and provide protected getters and setters. ~ Bhupesh On May 15, 2017 23:18, "Bhupesh Chawda" <bhup...@datatorrent.com> wrote: Actually Ajay discovered this when upgrading the core dependency test.

Re: Backward compatibility issue in 3.6.0 release

Actually Ajay discovered this when upgrading the core dependency test. One of the tests was failing. ~ Bhupesh On May 15, 2017 23:15, "Pramod Immaneni" wrote: I would suggest going with 1. Bhupesh how did you see this? Did it see it as part of working on an operator in

Re: Join support in Malhar

Created: https://issues.apache.org/jira/browse/APEXMALHAR-2488 ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Tue, May 9, 2017 at 8:47 PM, Bhupesh Chawda <b

[jira] [Created] (APEXMALHAR-2488) Simplify join support in Malhar

Bhupesh Chawda created APEXMALHAR-2488: -- Summary: Simplify join support in Malhar Key: APEXMALHAR-2488 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2488 Project: Apache Apex Malhar

Re: Proposal to upgrade Apex Core dependency in Malhar to 3.6

+1 ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Fri, May 12, 2017 at 10:32 AM, Priyanka Gugale <pri...@apache.org> wrote: > +1 to upgrade dependenc

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Thu, May 11, 2017 at 7:37 PM, Thomas Weise <t...@apache.org> wrote: > Usually batches are processed by different instances of a topolo

Re: Join support in Malhar

the JIRAs and PR if there is no response in a couple of days. ~ Bhupesh ​ ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Sat, May 6, 2017 at 11:07 PM, Thomas Weise <t...@apache.

Re: Join support in Malhar

implementation around? ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Fri, Apr 28, 2017 at 10:09 AM, Thomas Weise <t...@apache.org> wrote: > There is

Re: [VOTE] Apache Apex Core Release 3.6.0 (RC1)

+1 Checked the following: 1. Signatures and checksums okay 2. Build successful with tests 3. Presence of README.md, LICENSE, NOTICE and CHANGELOG.md files 4. Could launch pi demo successfully ~ Bhupesh ___ Bhupesh Chawda E: bhup

Re: PR merge policy

that the community has had a chance to review it, but there might still be cases where such undo commits need to be done. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org

Join support in Malhar

Hi Community, Currently the support for join in Malhar is little fuzzy for the end user. We have multiple implementations - 1. Join Impl 1 - Inner Join implementation, based on Managed state 2. Join Impl 2 - Merge operator, Windowed implementation, based on Spillable structures (based

Re: [Design] - Kudu Output Operator

Hi, Will be merging the PR - https://github.com/apache/apex-malhar/pull/486 soon unless there are any comments from the community. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com

Re: Checkstyle policies with auto generated code

Option 3 seems okay to me. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Mon, Apr 24, 2017 at 7:53 PM, Vlad Rozov <v.ro...@datatorrent.com> wrote:

Re: One Year Anniversary of Apache Apex

Congratulations to the community!! ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Tue, Apr 25, 2017 at 8:44 PM, Thomas Weise <t...@apache.org> wrote:

[jira] [Resolved] (APEXCORE-678) Shutdown of application should start from input nodes

[ https://issues.apache.org/jira/browse/APEXCORE-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXCORE-678. - Resolution: Fixed > Shutdown of application should start from input no

[jira] [Updated] (APEXCORE-654) Recovery window is not updated when Delay Operator is used along with Partitioned Operators

[ https://issues.apache.org/jira/browse/APEXCORE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXCORE-654: Attachment: ProblemDag.png > Recovery window is not updated when Delay Operator is u

Re: [VOTE] Apache Apex Malhar Release 3.7.0 (RC1)

+1 Verified signatures Verified build successfully Verified presence of README.md, CHANGELOG.md, LICENSE, NOTICE files ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org

[jira] [Commented] (APEXCORE-503) support KillException

[ https://issues.apache.org/jira/browse/APEXCORE-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944838#comment-15944838 ] Bhupesh Chawda commented on APEXCORE-503: - The graceful + abort shutdown for a dag has been

Re: Python Binding implementation for design review.

Ram, the URL seems to be split into multiple lines.. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Mon, Mar 27, 2017 at 6:51 PM, Munagala Ramanath &l

[jira] [Comment Edited] (APEXCORE-654) Recovery window is not updated when Delay Operator is used along with Partitioned Operators

[ https://issues.apache.org/jira/browse/APEXCORE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942151#comment-15942151 ] Bhupesh Chawda edited comment on APEXCORE-654 at 3/26/17 5:37 AM

[jira] [Commented] (APEXCORE-654) Recovery window is not updated when Delay Operator is used along with Partitioned Operators

[ https://issues.apache.org/jira/browse/APEXCORE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942151#comment-15942151 ] Bhupesh Chawda commented on APEXCORE-654: - The recovery window is not updated in cases like

[jira] [Assigned] (APEXCORE-654) Recovery window is not updated when Delay Operator is used along with Partitioned Operators

[ https://issues.apache.org/jira/browse/APEXCORE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda reassigned APEXCORE-654: --- Assignee: Bhupesh Chawda > Recovery window is not updated when Delay Opera

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

. For a windowed operator this would mean a Global window with a final watermark before the DAG is shut down. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Tue, Feb

[jira] [Updated] (APEXCORE-678) Shutdown of application should start from input nodes

[ https://issues.apache.org/jira/browse/APEXCORE-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXCORE-678: Description: Streaming container calls shutdown() for all nodes instead of just input

[jira] [Created] (APEXCORE-678) Shutdown of application should start from input nodes

Bhupesh Chawda created APEXCORE-678: --- Summary: Shutdown of application should start from input nodes Key: APEXCORE-678 URL: https://issues.apache.org/jira/browse/APEXCORE-678 Project: Apache Apex

[jira] [Updated] (APEXMALHAR-2451) Batch support for File I/O operators

[ https://issues.apache.org/jira/browse/APEXMALHAR-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXMALHAR-2451: --- Summary: Batch support for File I/O operators (was: Batch demarcation for File

[jira] [Updated] (APEXMALHAR-2449) Support for batch demarcation in input/output operators

[ https://issues.apache.org/jira/browse/APEXMALHAR-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXMALHAR-2449: --- Description: In order to support the demarcation of multiple batches

[jira] [Commented] (APEXCORE-575) Improve application relaunch time.

[ https://issues.apache.org/jira/browse/APEXCORE-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924130#comment-15924130 ] Bhupesh Chawda commented on APEXCORE-575: - [~tushargosavi] Can you please update the description

[jira] [Updated] (APEXMALHAR-2408) Issues in correctness of get() for key search in ManagedTimeStateImpl

[ https://issues.apache.org/jira/browse/APEXMALHAR-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXMALHAR-2408: --- Summary: Issues in correctness of get() for key search in ManagedTimeStateImpl

[jira] [Updated] (APEXMALHAR-2406) ManagedState Issues

[ https://issues.apache.org/jira/browse/APEXMALHAR-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda updated APEXMALHAR-2406: --- Description: Few bugs encountered while using ManagedState. Specific issues

[jira] [Commented] (APEXCORE-593) apex cli get-app-package-info could not retrieve properties defined in properties.xml

[ https://issues.apache.org/jira/browse/APEXCORE-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923801#comment-15923801 ] Bhupesh Chawda commented on APEXCORE-593: - [~vikram] Can you please add more details

Re: Watermark tuples in Apex

rks as per pre-defined logic and act as if it was received from upstream. The proposed watermarks could be used as implicit variants as well since "implicit watermark" is an orthogonal case. ~ Bhupesh _______ Bhupesh Chawda E: bhu

[jira] [Resolved] (APEXMALHAR-2350) The key and value stream should match with the bucket

[ https://issues.apache.org/jira/browse/APEXMALHAR-2350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXMALHAR-2350. Resolution: Fixed Fix Version/s: 3.7.0 > The key and value str

Re: APEXCORE-619 Recovery windowId in future during application relaunch.

My preference is also for option 3. It looks clean and simple to implement. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Fri, Mar 10, 2017 at 3:06 PM, Tushar

Watermark tuples in Apex

Hi All, Watermark tuples in Apex are very tightly coupled to event time processing. For this reason, usually they are modeled as having a timestamp. public interface WatermarkTuple { long getTimestamp(); } Even though, watermarks are meant for such time related processing, I think we should

[jira] [Created] (APEXCORE-660) Documentation for Control tuple support changes

Bhupesh Chawda created APEXCORE-660: --- Summary: Documentation for Control tuple support changes Key: APEXCORE-660 URL: https://issues.apache.org/jira/browse/APEXCORE-660 Project: Apache Apex Core

Re: APEXCORE-619 Recovery windowId in future during application relaunch.

by the stateless operator. Probably in this case at-least once is also not guaranteed? ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Thu, Mar 2, 2017 at 8:06 AM

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

firings every minute as well as drop any data which is say, 5 mins late. All this within a single day. ~ Bhupesh ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Tue, Feb 28, 2017

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Tue, Feb 28, 2017 at 12:28 AM, David Yan <david...@gmail.com> wrote: > I now see your rationale on putting the filename in the window. > As far as I understand, the reasons why

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

timestamp (which is just a sequence number) and that helps keep tuples from each file in a separate window. We could also implement a "timestampExtractor" interface to identify the timestamp (sequence number) for a file. ~ Bhupesh __________

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

to Discarding. Is this correct? I agree that (4. Final Watermark) can be done using Global window. ​~ Bhupesh​ ___ Bhupesh Chawda E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Mon, Feb 27, 2017 at 12

[jira] [Commented] (APEXMALHAR-2366) Apply BloomFilter to Bucket

[ https://issues.apache.org/jira/browse/APEXMALHAR-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882040#comment-15882040 ] Bhupesh Chawda commented on APEXMALHAR-2366: Hi [~brightchen] Sorry for commenting so

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

fferent partitions; unlike the file based watermarks where filename is the distinguishing property. ​ > > Once you make the changes to the operators, please also augment the > documentation and examples (in this case wordcount demo). > ​Sure.​ > Thanks, > Thomas > > >

Re: [Community Opinion] Use of Lombok Library

I think this could be a good way to improve readability of code in Malhar operators as well. This one however is MIT license. Any alternatives? ~ Bhupesh ___ Bhupesh Chawda Software Engineer E: bhup...@datatorrent.com | Twitter: @bhupeshsc

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

they too can be solved at the operator level where the input operators can play a role. We should keep the event time awareness logic out of the engine. ~ Bhupesh ___ Bhupesh Chawda Software Engineer E: bhup...@datatorrent.com | Twitter

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

. ___ Bhupesh Chawda Software Engineer E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Sat, Feb 18, 2017 at 10:24 PM, Thomas Weise <t...@apache.org> wrote: > Hi Bhupesh, > > I think this needs a generic wa

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

data to right files at the destination. ~ Bhupesh ___ Bhupesh Chawda Software Engineer E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Sat, Feb 18, 2017 at 12:26 PM, Amol Kekre

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

___ Bhupesh Chawda Software Engineer E: bhup...@datatorrent.com | Twitter: @bhupeshsc www.datatorrent.com | apex.apache.org On Thu, Feb 16, 2017 at 10:37 AM, Thomas Weise <t...@apache.org> wrote: > I don't think this should be designed based on a simplistic file > input-out

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

same purpose in each case. Please let me know your thoughts on this. ~ Bhupesh On Wed, Jan 18, 2017 at 12:22 AM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > Yes, this can be part of operator configuration. Given this, for a user to > define a batch application, would mea

Re: [DISCUSS] Custom Control Tuples Design

e or end batch in batch > use cases. > > Thanks > > On Tue, Jan 10, 2017 at 12:27 PM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Hi All, > > > > Based on some discussion here is what is planned for the propagation > > feature for

Re: Schema Discovery Support in Apex Applications

​I think the only concern for deciding the end-of-window-processing behavior for control tuples was that developers may, inadvertently mess up the idempotency of the operator. Even though we are trying to enforce this, it seems that even the emit of a control tuple (from the source) may not happen

Re: Shutdown of an Apex app

eeds to be signaled to the master and the master needs to > terminate. One way of doing it could be the stats listener. > > Thomas > > > On Tue, Jan 17, 2017 at 10:30 PM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > This JIRA is to stop the DAG in a crude manner,

Re: Shutdown of an Apex app

gt; stram can shutdown the entire app > > Thks > Amol > > > On Tue, Jan 17, 2017 at 11:05 PM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Yes Ajay, for a graceful shutdown, the data sent out should be processed. > > > > On Wed, Jan 18, 201

Re: Shutdown of an Apex app

do we plan to handle cases where 2 sub-DAGs merge to a single > operator somewhere downstream, and an operator in one of the sub-DAGs sends > ShutdownException. > > > Ajay > > On Wed, Jan 18, 2017 at 12:00 PM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > >

Re: Shutdown of an Apex app

ne Jira > > Opened for such functionality > > https://issues.apache.org/jira/browse/APEXCORE-503. > > > > - Tushar. > > > > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda > > <bhup...@datatorrent.com> wrote: > > > Hi

Re: Shutdown of an Apex app

r use > > cases were DAG needs to be shutdown after detecting some > > completion/error condition through the operator. We have one Jira > > Opened for such functionality > > https://issues.apache.org/jira/browse/APEXCORE-503. > > > > - Tushar. > >

Re: Shutdown of an Apex app

e Jira > Opened for such functionality > https://issues.apache.org/jira/browse/APEXCORE-503. > > - Tushar. > > > On Wed, Jan 18, 2017 at 11:45 AM, Bhupesh Chawda > <bhup...@datatorrent.com> wrote: > > Hi All, > > > > Currently we can shutdown an Apex a

Shutdown of an Apex app

Hi All, Currently we can shutdown an Apex app in the following ways: 1. Throw ShutdownException() from *all* the input operators 2. Use Apex CLI to shutdown an app using the YARN App Id I think we should have some way of shutting down an application from within an operator. It is not always true

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

o influence/configure it. > > Thomas > > > On Tue, Jan 17, 2017 at 5:03 AM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Hi Thomas, > > > > I am not sure that I completely understand your suggestion. Are you > > suggesting to broad

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

al watermark. > > Thanks, > Thomas > > > On Mon, Jan 16, 2017 at 1:02 AM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Yes, if the user needs to develop a batch application, then batch aware > > operators need to be used in the applicatio

Re: Schema Discovery Support in Apex Applications

+1 for the feature. ~ Bhupesh On Mon, Jan 16, 2017 at 5:09 PM, Chinmay Kolhatkar wrote: > Those are not really anonymous POJOs... The definition of POJO will be > known to user as based on that only upstream operator will convey the tuple > type the operator will be

Re: Contribution Process before PR

I agree. The discussion on requirements very much helps folks to understand the need for a feature and sometimes much more about related topics. From my own experience, the discussion thread on requirements before the implementation is what helps the design discussions and allows even people not

Re: Sharing jars among different Apex apps on cluster

> If stram changes the logical dag (runs only the sub-dag) on each runs, it > should be ok. > > Thks > Amol > > > On Mon, Jan 16, 2017 at 9:18 AM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Yes, I thought of that. That way it will be a single ap

Re: Sharing jars among different Apex apps on cluster

gt; On Mon, Jan 16, 2017 at 3:27 AM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Hi All, > > > > We have a use case where I need to launch a number of DAGs on the cluster > > one after the other in sequence programatically. > > > > We are u

Sharing jars among different Apex apps on cluster

Hi All, We have a use case where I need to launch a number of DAGs on the cluster one after the other in sequence programatically. We are using the StramAppLauncher and StramAppFactory classes to launch a DAG programatically on the cluster and adding any third party dependencies as part of the

Re: [DISCUSS] Proposal for adapting Malhar operators for batch use cases

l it make an impression on user that, if he has a batch usecase he has > to use batch aware operators only? If so, is that what we expect? I am not > aware of how do we implement batch scenario so this might be a basic > question. > > -Priyanka > > On Mon, Jan 16, 2017 at

[DISCUSS] Proposal for adapting Malhar operators for batch use cases

Hi All, While design / implementation for custom control tuples is ongoing, I thought it would be a good idea to consider its usefulness in one of the use cases - batch applications. This is a proposal to adapt / extend existing operators in the Apache Apex Malhar library so that it is easy to

Re: [DISCUSS] Custom Control Tuples Design

code to register the type with engine. And > platfoms needs to perform type check and develiver the tuples > accordingly. Instead operator developer could check the type of > incoming tuple and handle it as required. > > - Tushar. > > > > Thank you, > > > > Vlad >

Re: [DISCUSS] Custom Control Tuples Design

one both by API call and an > annotation. > > Thank you, > > Vlad > > > On 1/5/17 13:04, Bhupesh Chawda wrote: > >> Agreed Thomas. >> I was referring to the persona of the operator developer. The user of the >> operator would not be doing anything related

Re: [DISCUSS] Custom Control Tuples Design

2017 at 10:20 PM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > I think we all agree on the use case for selective propagation. The > question is about where to have the control - at the operator level or at > the port level. > > For this ability, we have the following

Re: [DISCUSS] Custom Control Tuples Design

t them on the output ports as desired. > > > > > > I don't see a use case for hybrid approaches? Maybe propagation does > not > > > need to be tied to ports at all, maybe just by annotation at the > operator > > > level? > > > > > > Thomas > > &

Re: [DISCUSS] Custom Control Tuples Design

No matter how many output ports there are. Output ports are there for an operator to emit new tuples, in the case you are discussing you don't emit new control tuples. Thomas On Wed, Jan 4, 2017 at 9:39 AM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > Hi Thomas, > > Are you

Re: [DISCUSS] Custom Control Tuples Design

s? The answer should be no. > > David > > On Sat, Dec 31, 2016 at 8:29 PM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Yes, that makes sense. > > We have following options: > > 1. Make the annotation false by default and force the user

[jira] [Resolved] (APEXMALHAR-2374) Recursive support for AbstractFileInputOperator

[ https://issues.apache.org/jira/browse/APEXMALHAR-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhupesh Chawda resolved APEXMALHAR-2374. Resolution: Fixed Fix Version/s: 3.7.0 > Recursive supp

Re: [DISCUSS] Custom Control Tuples Design

and tests. ~ Bhupesh On Sun, Jan 1, 2017 at 9:59 AM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > Yes, that makes sense. > We have following options: > 1. Make the annotation false by default and force the user to forward the > control tuples explicitly. > 2. Annotati

Re: [DISCUSS] Custom Control Tuples Design

d be able to > stop (consume without propagating) programmatically as well. > > Thanks > > On Thu, Dec 29, 2016 at 8:48 AM, Bhupesh Chawda <bhup...@datatorrent.com> > wrote: > > > Thanks Vlad, I am trying out the approach you mentioned regarding having &g

Re: [DISCUSS] Custom Control Tuples Design

r control aware sinks will extend > from. > > Thank you, > > Vlad > > > On 12/23/16 06:24, Bhupesh Chawda wrote: > >> Hi Vlad, >> >> Thanks for the pointer on delegating the wrapping of the user tuple to the >> control port. I was trying this o

[jira] [Commented] (APEXMALHAR-2368) JDBCPollInput operator reads extra records when 1.5M records are added to a blank input table

[ https://issues.apache.org/jira/browse/APEXMALHAR-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1553#comment-1553 ] Bhupesh Chawda commented on APEXMALHAR-2368: [~Hitesh_] Can you please give some

Re: [DISCUSS] Custom Control Tuples Design

rt? Can't it be delegated to a > Sink by introducing new putControlTuple method? > > Thank you, > > Vlad > > > On 12/21/16 22:10, Bhupesh Chawda wrote: > >> Hi Vlad, >> >> The problem in using the Tuple class as the wrapper is that the Ports >> belong to t

Re: [DISCUSS] Custom Control Tuples Design

understand how adding more methods to the Default implementation >> will help with early error detection unless application or operator that >> relies on the custom control tuple functionality explicitly checks for the >> platform version at run-time or tries to emit a control tuple just to chec

Re: [DISCUSS] Custom Control Tuples Design

n or operator that > relies on the custom control tuple functionality explicitly checks for the > platform version at run-time or tries to emit a control tuple just to check > that such functionality is supported by the platform. > > Thank you, > > Vlad > > On 12/21/16 04:58

  1   2   >