[jira] [Commented] (APEXMALHAR-2411) Avoid isreplaystate variable, incorporate logic in activate() and replay() for Kinesis Input Operator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871245#comment-15871245
 ] 

ASF GitHub Bot commented on APEXMALHAR-2411:


GitHub user deepak-narkhede opened a pull request:

https://github.com/apache/apex-malhar/pull/555

APEXMALHAR-2411 Avoid isreplaystate variable, incorporate logic in 
activate() and replay() for Kinesis Input Operator

Avoid isreplaystate variable, incorporate logic in activate() and replay() 
for Kinesis Input Operator.

Unit test testRecoveryAndIdempotency() passed with the fix initially it 
used to fail.
Also tested with Kinesis to S3 app.

 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/deepak-narkhede/apex-malhar APEXMALHAR-2411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #555


commit 0c0a8cf690742fbec3b98d32641bf6bdbaf9a166
Author: deepak-narkhede 
Date:   2017-02-17T06:13:05Z

APEXMALHAR-2411 Avoid isreplaystate variable, incorporate logic in 
activate() and replay() for Kinesis Input Operator




> Avoid isreplaystate variable, incorporate logic in activate() and replay() 
> for Kinesis Input Operator
> -
>
> Key: APEXMALHAR-2411
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2411
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Deepak Narkhede
>Assignee: Deepak Narkhede
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-malhar pull request #555: APEXMALHAR-2411 Avoid isreplaystate variable,...

[deepak-narkhede]

GitHub user deepak-narkhede opened a pull request:

https://github.com/apache/apex-malhar/pull/555

APEXMALHAR-2411 Avoid isreplaystate variable, incorporate logic in 
activate() and replay() for Kinesis Input Operator

Avoid isreplaystate variable, incorporate logic in activate() and replay() 
for Kinesis Input Operator.

Unit test testRecoveryAndIdempotency() passed with the fix initially it 
used to fail.
Also tested with Kinesis to S3 app.

 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/deepak-narkhede/apex-malhar APEXMALHAR-2411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #555


commit 0c0a8cf690742fbec3b98d32641bf6bdbaf9a166
Author: deepak-narkhede 
Date:   2017-02-17T06:13:05Z

APEXMALHAR-2411 Avoid isreplaystate variable, incorporate logic in 
activate() and replay() for Kinesis Input Operator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXMALHAR-2409) Improve PojoInnerJoin accumulation to emit a POJO instead of Map

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871231#comment-15871231
 ] 

ASF GitHub Bot commented on APEXMALHAR-2409:


GitHub user chinmaykolhatkar opened a pull request:

https://github.com/apache/apex-malhar/pull/554

APEXMALHAR-2409 PojoInnerJoin accumulation to emit a POJO

@tushargosavi Please review and merge.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chinmaykolhatkar/apex-malhar 
APEXMALHAR-2409_PojoInnerJoinObject

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #554


commit a00a4c78986837725d6c129f8b7aaa182134d500
Author: Chinmay Kolhatkar 
Date:   2017-02-17T06:05:30Z

APEXMALHAR-2409 PojoInnerJoin accumulation to emit a POJO




> Improve PojoInnerJoin accumulation to emit a POJO instead of Map
> 
>
> Key: APEXMALHAR-2409
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2409
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 3.6.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Currently POJO Inner Join accumulation emits a Map. It should emit a POJO.
> The parameter to the accumulation should accept POJO list mapping from input 
> to output variables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-malhar pull request #554: APEXMALHAR-2409 PojoInnerJoin accumulation to...

[chinmaykolhatkar]

GitHub user chinmaykolhatkar opened a pull request:

https://github.com/apache/apex-malhar/pull/554

APEXMALHAR-2409 PojoInnerJoin accumulation to emit a POJO

@tushargosavi Please review and merge.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chinmaykolhatkar/apex-malhar 
APEXMALHAR-2409_PojoInnerJoinObject

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #554


commit a00a4c78986837725d6c129f8b7aaa182134d500
Author: Chinmay Kolhatkar 
Date:   2017-02-17T06:05:30Z

APEXMALHAR-2409 PojoInnerJoin accumulation to emit a POJO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXMALHAR-2414) Improve performance of PojoInnerJoin accum by using PojoUtils

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2414:
-

 Summary: Improve performance of PojoInnerJoin accum by using 
PojoUtils
 Key: APEXMALHAR-2414
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2414
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Reporter: Chinmay Kolhatkar


Currently PojoInnerJoin accumulation uses java reflection to deal with objects. 
Use PojoUtils instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2415) Enable PojoInnerJoin accum to allow multiple keys for join purpose

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2415:
-

 Summary: Enable PojoInnerJoin accum to allow multiple keys for 
join purpose
 Key: APEXMALHAR-2415
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2415
 Project: Apache Apex Malhar
  Issue Type: Sub-task
Affects Versions: 3.6.0
Reporter: Chinmay Kolhatkar


Enable PojoInnerJoin accum to allow multiple keys for join purpose



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (APEXMALHAR-2409) Improve PojoInnerJoin accumulation to emit a POJO instead of Map

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar updated APEXMALHAR-2409:
--
Issue Type: Sub-task  (was: Improvement)
Parent: APEXMALHAR-2413

> Improve PojoInnerJoin accumulation to emit a POJO instead of Map
> 
>
> Key: APEXMALHAR-2409
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2409
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 3.6.0
>Reporter: Chinmay Kolhatkar
>Assignee: Chinmay Kolhatkar
>
> Currently POJO Inner Join accumulation emits a Map. It should emit a POJO.
> The parameter to the accumulation should accept POJO list mapping from input 
> to output variables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2413) Improve PojoInnerJoin Accumulation

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2413:
-

 Summary: Improve PojoInnerJoin Accumulation
 Key: APEXMALHAR-2413
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2413
 Project: Apache Apex Malhar
  Issue Type: Improvement
  Components: algorithms
Affects Versions: 3.6.0
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


This contains a series of subtasks:
1. Enable PojoJoinInner accumulation to emit POJO instead of Map
2. Improve performance of PojoInnerJoin accumulation by using PojoUtils instead 
of java reflection
3. Enable PojoInnerJoin accumulation to accept group of keys for join instead 
of a single key.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (APEXMALHAR-2411) Avoid isreplaystate variable, incorporate logic in activate() and replay() for Kinesis Input Operator

[Deepak Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Narkhede updated APEXMALHAR-2411:

Summary: Avoid isreplaystate variable, incorporate logic in activate() and 
replay() for Kinesis Input Operator  (was: Avoid consumer start while replay 
state in KinesisInputOperator)

> Avoid isreplaystate variable, incorporate logic in activate() and replay() 
> for Kinesis Input Operator
> -
>
> Key: APEXMALHAR-2411
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2411
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Deepak Narkhede
>Assignee: Deepak Narkhede
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2412) Provide emitTuple overriding functionality for user in kinesis Input operator

[Deepak Narkhede (JIRA)]

Deepak Narkhede created APEXMALHAR-2412:
---

 Summary: Provide emitTuple overriding functionality for user in 
kinesis Input operator
 Key: APEXMALHAR-2412
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2412
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Deepak Narkhede






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXMALHAR-2411) Avoid consumer start while replay state in KinesisInputOperator

[Deepak Narkhede (JIRA)]

Deepak Narkhede created APEXMALHAR-2411:
---

 Summary: Avoid consumer start while replay state in 
KinesisInputOperator
 Key: APEXMALHAR-2411
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2411
 Project: Apache Apex Malhar
  Issue Type: Improvement
Reporter: Deepak Narkhede
Assignee: Deepak Narkhede






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-624) Shutdown does not work because of incorrect logic in the AppMaster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871040#comment-15871040
 ] 

ASF GitHub Bot commented on APEXCORE-624:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/458


> Shutdown does not work because of incorrect logic in the AppMaster
> --
>
> Key: APEXCORE-624
> URL: https://issues.apache.org/jira/browse/APEXCORE-624
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Sanjay M Pujare
>Assignee: Sanjay M Pujare
>Priority: Minor
>
> com.datatorrent.stram.StreamingAppMasterService.execute() calculates 
> numRequestedContainers incorrectly in some cases (e.g. RM container 
> allocation failure) which prevents an application from shutting down when it 
> is requested externally. An example is where we ask RM to remove previous 
> container allocation request (where the count should be decremented but is 
> NOT) and add a new one (where the count should be and IS incremented). 
> Another example is the "alreadyAllocated" case where we release the container 
> and still increment numRequestedContainers which seems wrong. 
> This bug is showing up in multiple Apex deployments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-core pull request #458: APEXCORE-624 decrement unallocated containers a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/458


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXMALHAR-2410) KafkaSinglePortExactlyOnceOutputOperator fails if an operator crashes in the very first window

[Oliver Winke (JIRA)]

Oliver Winke created APEXMALHAR-2410:


 Summary: KafkaSinglePortExactlyOnceOutputOperator fails if an 
operator crashes in the very first window
 Key: APEXMALHAR-2410
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2410
 Project: Apache Apex Malhar
  Issue Type: Bug
Reporter: Oliver Winke
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (APEXCORE-604) Extend DAG API to allow accessing DAG objects.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869809#comment-15869809
 ] 

ASF GitHub Bot commented on APEXCORE-604:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/447


> Extend DAG API to allow accessing DAG objects.
> --
>
> Key: APEXCORE-604
> URL: https://issues.apache.org/jira/browse/APEXCORE-604
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
>
> I am planing extend the DAG api to export internals of the DAG.
> Current DAG does not provide a way to get the list of operators and
> streams with their attributes. Also streamMeta does not provide
> API to access end ports.
> This type of information is needed when external translator (like
> Samoa) are constructing the DAG,
> and we need to apply some transformation or configure the DAG before
> it is run. Planing to extend
> DAG API with following.
> InputPortMeta
> public Operator.InputPort getPortObject();
> OutputPortMeta
> public Operator.OutputPort getPortObject();
> streamMeta
> public  T getSource();
> public  Collection getSinks();
> DAG
> public abstract  Collection getOperators();
> public abstract  Collection getStreams();



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-core pull request #447: APEXCORE-604 extend DAG API to get operators an...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/447


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: PojoInnerJoin Accumulation emitting Map

[Chinmay Kolhatkar]

APEXMALHAR-2409 Created.

I'll create a PR soon.

On Wed, Feb 15, 2017 at 10:31 PM, Chinmay Kolhatkar  wrote:

> Thanks guys. I'll create a jira and create a PR for it soon.
>
> Ajay, btw this is not much related to Schema discovery design... This is
> for windowed operator.
>
> - Chinmay.
>
> On 15 Feb 2017 10:11 p.m., "Amol Kekre"  wrote:
>
>> yes it should be POJO
>>
>> Thks
>> Amol
>>
>> *Join us at Apex Big Data World-San Jose
>> , April 4, 2017!*
>> [image: http://www.apexbigdata.com/san-jose-register.html]
>> 
>>
>> On Wed, Feb 15, 2017 at 7:34 AM, AJAY GUPTA  wrote:
>>
>> > Yes, it should be emitting a POJO. This POJO can then be further used to
>> > join with a third POJO stream, thus behaving like a DB join.
>> >
>> > It would be best if we can incorporate this into Schema Discovery
>> design.
>> >
>> >
>> > Ajay
>> >
>> > On Wed, 15 Feb 2017 at 5:30 PM, Chinmay Kolhatkar 
>> > wrote:
>> >
>> > Dear Community,
>> >
>> > Currently PojoInnerJoin accumulation is accepting 2 POJOs but emitting a
>> > Map.
>> >
>> > I think it should be emitting POJO instead of Map.
>> >
>> > Please share your thoughts about this.
>> >
>> > Thanks,
>> > Chinmay.
>> >
>>
>

[jira] [Created] (APEXMALHAR-2409) Improve PojoInnerJoin accumulation to emit a POJO instead of Map

[Chinmay Kolhatkar (JIRA)]

Chinmay Kolhatkar created APEXMALHAR-2409:
-

 Summary: Improve PojoInnerJoin accumulation to emit a POJO instead 
of Map
 Key: APEXMALHAR-2409
 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2409
 Project: Apache Apex Malhar
  Issue Type: Improvement
  Components: streams
Affects Versions: 3.6.0
Reporter: Chinmay Kolhatkar
Assignee: Chinmay Kolhatkar


Currently POJO Inner Join accumulation emits a Map. It should emit a POJO.

The parameter to the accumulation should accept POJO list mapping from input to 
output variables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)