from:"sandeshh"

[GitHub] apex-malhar pull request #633: APEXMALHAR-2506 Added the monitoring to Kafka...

2017-06-09 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-malhar/pull/633

APEXMALHAR-2506 Added the monitoring to Kafka Consumer threads, if onâ¦

â¦e of them dies then the Kafka operator is killed so that it can recover 
from the proper state.

@PramodSSImmaneni please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2506

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #633






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #543: APEXCORE-743 Added timeout for the Container ki...

2017-06-09 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/543

APEXCORE-743 Added timeout for the Container kill request sent to NM.

@PramodSSImmaneni @vrozov please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-743

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/543.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #543


commit 501dfa47517f94aa35d60c4e22ec825e2c99fa27
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-06-01T23:28:56Z

APEXCORE-743 Added timeout for the Container kill request sent to NM.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #465: Dont Merge/Review yet APEXCORE-632 BufferServ...

2017-04-17 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/465


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #462: APEXCORE-630 Removing FastDataList as it is not...

2017-04-17 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/462


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #483: APEXCORE-663 During restart fallback to old way...

2017-03-07 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/483

APEXCORE-663 During restart fallback to old way of constructing the lâ¦

â¦ist of containers as YARN return empty list of containers

@vrozov @tweise please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-663

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/483.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #483


commit 3ca87d0abfe8cab9d87c317defb57ae19dd61d26
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-03-08T07:43:24Z

APEXCORE-663 During restart fallback to old way of constructing the list of 
containers as YARN return empty list of containers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #560: APEXMALHAR-2422 Flushing the data to HDFS nee...

2017-02-23 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-malhar/pull/560


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #560: APEXMALHAR-2422 Flushing the data to HDFS nee...

2017-02-23 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-malhar/pull/560


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #560: APEXMALHAR-2422 Flushing the data to HDFS nee...

2017-02-23 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/560

APEXMALHAR-2422 Flushing the data to HDFS needs a separate flag for iâ¦

â¦t to work.

@otterc , @tushargosavi please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2422

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/560.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #560


commit 7682d368c78a4aecc84f1675ffe48fe2efa13421
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-02-23T20:51:48Z

APEXMALHAR-2422 Flushing the data to HDFS needs a separate flag for it to 
work.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #560: APEXMALHAR-2422 Flushing the data to HDFS nee...

2017-02-23 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/560

APEXMALHAR-2422 Flushing the data to HDFS needs a separate flag for iâ¦

â¦t to work.

@otterc , @tushargosavi please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2422

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/560.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #560


commit 7682d368c78a4aecc84f1675ffe48fe2efa13421
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-02-23T20:51:48Z

APEXMALHAR-2422 Flushing the data to HDFS needs a separate flag for it to 
work.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #560: APEXMALHAR-2422 Flushing the data to HDFS nee...

2017-02-23 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-malhar/pull/560

APEXMALHAR-2422 Flushing the data to HDFS needs a separate flag for iâ¦

â¦t to work.

@otterc , @tushargosavi please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2422

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/560.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #560


commit 7682d368c78a4aecc84f1675ffe48fe2efa13421
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-02-23T20:51:48Z

APEXMALHAR-2422 Flushing the data to HDFS needs a separate flag for it to 
work.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #465: APEXCORE-632 BufferServer unit tests moving to ...

2017-02-06 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/465

APEXCORE-632 BufferServer unit tests moving to Junit

1. Move the unit tests to Junit from testNg
2. Fix the failing test cases
3. Enable the test cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-632

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/465.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #465


commit 89de5f8616cfe8f61dd2ea77681c78c6e977f980
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-02-02T18:49:44Z

APEXCORE-632 BufferServer unit tests moving to Junit

1. Move the unit tests to Junit from testNg
2. Fix the failing test cases
3. Enable the test cases




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #453: Review only APEXCORE-259 Directly write to Da...

2017-02-01 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/453


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #462: APEXCORE-630 Removing FastDataList as it is not...

2017-02-01 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/462

APEXCORE-630 Removing FastDataList as it is not complete and not deveâ¦

â¦loped, so no reason to keep it

@vrozov please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-630

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #462


commit f90e1f829d904154d9c577104ed5b865d8da5dda
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-02-01T20:21:19Z

APEXCORE-630 Removing FastDataList as it is not complete and not developed, 
so no reason to keep it




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #453: Review only APEXCORE-259 Directly write to Da...

2017-01-23 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/453

*Review only* APEXCORE-259 Directly write to DataList

Instead of the loop back between the Output port and the buffer server,  
tuples are directly written to DataList. This improves the tuples write to 
BufferServer, but exposing the bottleneck between downStream and BufferServer, 
which in turn mean too much spooling.

This PR along with the following PRs improves the performance to great 
extent.

https://github.com/apache/apex-core/pull/445
https://github.com/apache/apex-core/pull/436

@vrozov and @PramodSSImmaneni please see.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/453.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #453


commit 66a3f756f90ef27df00497d1584f53ece0b74446
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2017-01-22T07:44:25Z

APEXCORE-259 Directly write to DataList




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #443: APEXCORE-493 checkpoint purging from Streaming ...

2017-01-23 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/443


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #443: APEXCORE-493 checkpoint purging from streaming ...

2017-01-05 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/443

APEXCORE-493 checkpoint purging from streaming container



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-493

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #443


commit 832580af99172f3625f1071ba74e1b0534eaff61
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-11-08T22:43:54Z

APEXCORE-493 checkpoint purging from streaming container




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #437: APEXCORE-471 Reissue of the resource was failin...

2016-12-24 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/437

APEXCORE-471 Reissue of the resource was failing for the BlackListBasâ¦

â¦ed scheduler, it happened because RequestedResource was never empty. 
Issue is fixed after making the operation similar to that of 
ResourceRequestHandler with the handling for Blacklist

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-471

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #437


commit 693cb30541531004e6c2f11b68e099763402a6c7
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-12-25T07:25:31Z

APEXCORE-471 Reissue of the resource was failing for the BlackListBased 
scheduler, it happened because RequestedResource was never empty. Issue is 
fixed after making the operation similar to that of ResourceRequestHandler with 
the handling for Blacklist




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #430: APEXCORE-584 Exposed the attribute to blacklist...

2016-12-07 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/430

APEXCORE-584 Exposed the attribute to blacklist the nodes for the resâ¦

â¦ource allocation purpose

@PramodSSImmaneni please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-584

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #430


commit 4eb5817009cd98fb9f7b20fdc4ffd493f980deaa
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-12-07T22:51:35Z

APEXCORE-584 Exposed the attribute to blacklist the nodes for the resource 
allocation purpose




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #492: APEXMALHAR-2298 Making the KafkaExactlyOnceOu...

2016-11-10 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-malhar/pull/492

APEXMALHAR-2298 Making the KafkaExactlyOnceOutputOperator take generiâ¦

â¦c object as input.

@siyuanh please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2298

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #492


commit 452cb37246e005da49935ab85a0e7efc131ead4c
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-11-08T19:25:16Z

APEXMALHAR-2298 Making the KafkaExactlyOnceOutputOperator take generic 
object as input.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #416: Review only APEXCORE-426 Reuse the running co...

2016-11-04 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/416

*Review only* APEXCORE-426 Reuse the running container, when the Stram 
restarts.

@tweise & @vrozov  please review

If you guys are fine with the approach, I will work on unit tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-426

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/416.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #416






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #407: APEXCORE-476 toString() method was returning wr...

2016-10-05 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/407

APEXCORE-476 toString() method was returning wrong window ids, fixingâ¦

â¦ this, as the correct information is useful during debugging

@vrozov please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-476

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/407.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #407


commit 4a4aebb7c469fd09f6ad569dee124aabaf9b51e4
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-10-05T15:00:48Z

APEXCORE-476 toString() method was returning wrong window ids, fixing this, 
as the correct information is useful during debugging




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #406: APEXCORE-475 Making the application type of ape...

2016-10-05 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/406

APEXCORE-475 Making the application type of apex applications as Apacâ¦

â¦heApex and we will still keep DataTorrent as the deprecated tag.

@tweise please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-475

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/406.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #406


commit db60103e02f220a9fd424a97183ee8d8194d4479
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-10-05T14:44:23Z

APEXCORE-475 Making the application type of apex applications as ApacheApex 
and we will still keep DataTorrent as the deprecated tag.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #405: APEXCORE-505 Heartbeat loop was blocked waiting...

2016-10-04 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-core/pull/405

APEXCORE-505 Heartbeat loop was blocked waiting for operator activatiâ¦

â¦on, the reason for this is that Stream activation(Only 
BufferServerSubscriber and WindowGenerator) waits for operator activation in a 
heartbeat thread. There is no need to have this synchronization, as Tuples are 
pulled from the queues by the operators.

@vrozov @tweise please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-505

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #405


commit 596b996b1aa694bb9a0924f9094ef688df23678e
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-10-05T01:29:57Z

APEXCORE-505 Heartbeat loop was blocked waiting for operator activation, 
reason for this is that Stream activation(Only BufferServerSubscriber and 
WindowGenerator) waits for operator activation in heartbeat thread. After 
analysis and sanity testing, we don't see the need to have the synchronization 
between operator and stream activation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #405: APEXCORE-505 Heartbeat loop was blocked waiting...

2016-10-04 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/405

APEXCORE-505 Heartbeat loop was blocked waiting for operator activatiâ¦

â¦on, the reason for this is that Stream activation(Only 
BufferServerSubscriber and WindowGenerator) waits for operator activation in a 
heartbeat thread. There is no need to have this synchronization, as Tuples are 
pulled from the queues by the operators.

@vrozov @tweise please review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-505

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #405


commit 596b996b1aa694bb9a0924f9094ef688df23678e
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-10-05T01:29:57Z

APEXCORE-505 Heartbeat loop was blocked waiting for operator activation, 
reason for this is that Stream activation(Only BufferServerSubscriber and 
WindowGenerator) waits for operator activation in heartbeat thread. After 
analysis and sanity testing, we don't see the need to have the synchronization 
between operator and stream activation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #400: APEXCORE-543 ContainerInfo will contain Map of ...

2016-09-30 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/400

APEXCORE-543 ContainerInfo will contain Map of operators, with Key reâ¦

â¦presenting Id and Value representing name.

@davidyan74 please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-543

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit 6f92a80244f8d1545a58367bdf676cb82da28903
Author: Sandesh Hegde <sandesh.he...@gmail.com>
Date:   2016-09-30T22:34:13Z

APEXCORE-543 ContainerInfo will contain Map of operators, with Key 
representing Id and Value representing name.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #395: [WIP] APEXCORE-474 In M*1 case, deploy the unif...

2016-09-20 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/395

[WIP] APEXCORE-474 In M*1 case, deploy the unifier in the same container 
asâ¦

â¦ downstream

Opening this, work in progress, PR to get early feedback. Only changes I am 
doing is for unit tests.

There are another 4 failing tests.

@tweise and @PramodSSImmaneni Please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #395


commit 91a8470b7e465ba767372956d3311238d720c23e
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-09-20T22:56:23Z

APEXCORE-474 In M*1 case, deploy the unifier in the same container as 
downstream




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #391: APEXCORE-533 layering of the properties in case...

2016-09-14 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/391

APEXCORE-533 layering of the properties in case of config apps where â¦

â¦failing, because of the order in which they are processed, now fixed

@davidyan74 please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-533

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #391


commit 7fd69250f0ff42c956e5234129293adc3152cb51
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-09-15T00:58:26Z

APEXCORE-533 layering of the properties in case of config apps where 
failing, because of the order in which they are processed, now fixed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #369: Review only [APEXCORE-498] Savepoint feature

2016-09-13 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/369


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #378: APEXCORE-474

2016-08-31 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/378

APEXCORE-474

https://issues.apache.org/jira/browse/APEXCORE-474

Creating a separate container for the last unifier is not required. It only 
changes the default behavior, users can still create the separate unifiers.

@PramodSSImmaneni & @tweise please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/378.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #378


commit 3c5bc858967b0dfbebc5422aaaba93fbc7e7a3c8
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-08-31T18:26:12Z

changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #373: APEXCORE-475 1st part, as discussed in the jira...

2016-08-24 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/373

APEXCORE-475 1st part, as discussed in the jira. This will make the câ¦

â¦lient the forward compatible, then in the next release we can change the 
type to of the applicationt to ApacheApex

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-475

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/373.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #373


commit cf5db3367f5e31a55e86b77d68a2e0528dbe5fee
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-08-24T14:12:38Z

APEXCORE-475 1st part, as discussed in the jira. This will make the client 
the forward compatible, then in the next release we can change the type to of 
the applicationt to ApacheApex




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #358: Review only [APEXCORE-482]Make the downstream...

2016-08-23 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/358


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #372: APEXCORE-513 Reduce the log level

2016-08-22 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/372

APEXCORE-513 Reduce the log level



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-513

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/372.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #372


commit 6063a955ff4141f81042a03cc5cdfc0e62096520
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-08-23T03:38:35Z

APEXCORE-513 Reduce the log level




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #371: APEXCORE-512 While processing the "-exactMatch"...

2016-08-22 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-core/pull/371

APEXCORE-512 While processing the "-exactMatch" flag, immutable list â¦

â¦was required. Made the changes to unit test as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-512

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/371.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #371


commit 8c1cd987da1fe59eeae4bc3503846bddf589648d
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-08-22T18:47:32Z

APEXCORE-512 While processing the "-exactMatch" flag, immutable list was 
required.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #371: APEXCORE-512 While processing the "-exactMatch"...

2016-08-22 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/371

APEXCORE-512 While processing the "-exactMatch" flag, immutable list â¦

â¦was required. Made the changes to unit test as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-512

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/371.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #371


commit 8c1cd987da1fe59eeae4bc3503846bddf589648d
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-08-22T18:47:32Z

APEXCORE-512 While processing the "-exactMatch" flag, immutable list was 
required.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #369: Review only [APEXCORE-498] Savepoint feature

2016-08-15 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/369

*Review only* [APEXCORE-498] Savepoint feature

This enables the following feature,

a. Store the savepoint of the running app in the location provided
apexCli> savepoint   [-overWrite]

b. apexCli> launch -savePoint 

Limitations:
 -> Custom Storage Agent used for checkpoints is not supported, ( v2 may be 
? )
-> Existing recovery window is used in the savepoint, so taking savepoint 
is not possible before the 1st recovery window

TODO: Unit Tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-498

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/369.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #369


commit cc01813c1d103c3f9471018b67731bd8163aa18f
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-08-15T06:20:12Z

APEXCORE-498




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #361: APEXCORE-496 make operator name available to St...

2016-08-09 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/361#discussion_r74098254
  
--- Diff: api/src/main/java/com/datatorrent/api/StatsListener.java ---
@@ -115,6 +115,28 @@
 List getOperatorResponse();
   }
 
+  /**
+   * An interface to the DAG. Stats listener can get information about
+   * operator or other elements in the DAG through this interface. 
currerntly
+   * we only provide method to extract the operator name based on the 
physical
+   * id of the operator. In future more methods can be added.
+   *
+   */
--- End diff --

For the batch scheduling support, having the DAG view available for the 
operator will be useful. Users should be able to extract the information from 
the DAG view. Exposing the method for every information may not be ideal.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

2016-08-04 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/360#discussion_r73584686
  
--- Diff: engine/src/test/java/com/datatorrent/stram/cli/ApexCliTest.java 
---
@@ -180,4 +188,40 @@ public void 
testLaunchAppPackagePrecedenceWithConfigPackage() throws Exception
 Assert.assertEquals("app-default", props.get("dt.test.5"));
 Assert.assertEquals("package-default", props.get("dt.test.6"));
   }
+
+  @Test
+  public void testAppFromOnlyConfigPackage() throws Exception
+  {
+ApexCli.LaunchCommandLineInfo commandLineInfo =
+ApexCli.getLaunchCommandLineInfo(new String[]{"-conf", 
configFile.getAbsolutePath(), appFile.getAbsolutePath(), "-useConfigApps", 
"exclusive"});
+
+ApexCli apexCli = new ApexCli();
+apexCli.init();
+
+Assert.assertEquals("configApps", "exclusive", 
commandLineInfo.useConfigApps);
+
+apexCli.getLaunchAppPackageArgs(ap, cp, commandLineInfo, new 
ConsoleReader());
+
+Assert.assertEquals(ap.getApplications().size(), 1);
+  }
+
+  @Test
+  public void testMergeAppFromConfigAndAppPackage() throws Exception
+  {
+ApexCli.LaunchCommandLineInfo commandLineInfo =
+ApexCli.getLaunchCommandLineInfo(new String[]{"-conf", 
configFile.getAbsolutePath(), appFile.getAbsolutePath(), "-useConfigApps", 
"inclusive"});
+
+Assert.assertEquals("configApps", "inclusive", 
commandLineInfo.useConfigApps);
+
+ApexCli apexCli = new ApexCli();
+apexCli.init();
+
+try {
+  apexCli.getLaunchAppPackageArgs(ap, cp, commandLineInfo, new 
ConsoleReader());
+} catch (Throwable throwable ) {
--- End diff --

Feature, you mentioned is already present. Exception is also an existing 
behaviour, if the apex cli is not connected to console it throws an exception. 
In the unit test there is no console.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core issue #360: APEXCORE-495 supporting apps in config package.

2016-08-02 Thread sandeshh

Github user sandeshh commented on the issue:

https://github.com/apache/apex-core/pull/360
  
@davidyan74 Please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #364: APEXCORE-448 Made operator name available in op...

2016-08-01 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/364#discussion_r73067721
  
--- Diff: api/src/main/java/com/datatorrent/api/Context.java ---
@@ -328,6 +328,8 @@
  */
 int getWindowsFromCheckpoint();
 
+String getOperatorName();
--- End diff --

+1 for getName. Calls are inside the classes starting with Operator*, 
methods do not need to contain word *operator*.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

2016-07-28 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/360#discussion_r72670932
  
--- Diff: 
engine/src/test/java/com/datatorrent/stram/support/StramTestSupport.java ---
@@ -157,21 +160,42 @@ public static void 
waitForWindowComplete(OperatorContext nodeCtx, long windowId)
*/
   public static File createAppPackageFile()
   {
+buildOperators();
+
 final String version = System.getProperty("apex.version");
 final List params = new LinkedList<>();
 params.add("clean");
-params.add("package");
+params.add("install");
--- End diff --

will make it package


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

2016-07-28 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/360#discussion_r72670717
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/client/ConfigPackage.java ---
@@ -71,6 +72,7 @@
 
   private final Map<String, String> properties = new TreeMap<>();
   private final Map<String, Map<String, String>> appProperties = new 
TreeMap<>();
+  private List applications = new ArrayList<>();
--- End diff --

Yes, it can be final


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #360: APEXCORE-495 supporting apps in config package.

2016-07-28 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/360#discussion_r72670234
  
--- Diff: engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java ---
@@ -3850,6 +3908,9 @@ private Option add(Option opt)
 final Option exactMatch = add(new Option("exactMatch", "Only consider 
applications with exact app name"));
 final Option queue = add(OptionBuilder.withArgName("queue 
name").hasArg().withDescription("Specify the queue to launch the 
application").create("queue"));
 final Option force = add(new Option("force", "Force launch the 
application. Do not check for compatibility"));
+final Option onlyConfigApps = add(new Option("onlyConfigApps", "Only 
use the apps present in the config package."));
+final Option skipConfigApps = add(new Option("skipConfigApps", "Skip 
the apps present in the config package."));
+
--- End diff --

Yes, that is better. I will change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core issue #358: Review only [APEXCORE-482]Make the downstream operat...

2016-07-22 Thread sandeshh

Github user sandeshh commented on the issue:

https://github.com/apache/apex-core/pull/358
  
@tweise I will do the benchmarking. But at the outset, It skips the queue 
between default unifier and the downstream, that should improve performance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #360: review only APEXCORE-495 supporting apps in c...

2016-07-21 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/360

*review only* APEXCORE-495 supporting apps in config package.

1. Added support to store apps in the config file
2. By default apps from the config and app package gets merged (During 
conflict Config apps overwrite the AppPackage apps)
3. New option during launch, "onlyConfigApps", which will only show config 
apps
4. Assumes that the operators are not part of App jars.

TODO: Add unit test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-495

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #360


commit f75d747df24c146110e1e276521f16782be61762
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-07-21T22:18:29Z

APEXCORE-495
1. Added support to store apps in the config file
2. By default apps from the config and app package gets merged (During 
conflict Config apps overwrite the AppPackage apps)
3. New option during launch, "onlyConfigApps", which will only show config 
apps
4. Assumes that the operators are not part of App jars.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-12 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-malhar/pull/298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-12 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/298

[APEXMALHAR-2086] Kafka output operator: 0.9.0

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 3cf9c7ea9d2ba8d62586923d202fb5207c7dc45c
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka 0.9.0 output operators and unit tests.

1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
 Key in the Kafka message is used by the operator to track the tuples 
written by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-07-11 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/350


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #357: APEXCORE-488: Issues in SSL communication with ...

2016-07-11 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/357#discussion_r70355804
  
--- Diff: engine/src/main/java/com/datatorrent/stram/client/StramAgent.java 
---
@@ -204,7 +204,14 @@ private UriBuilder 
getStramWebURIBuilder(WebServicesClient webServicesClient, St
 if (info != null) {
   //ws = wsClient.resource("http://; + 
info.appMasterTrackingUrl).path(WebServices.PATH).path(info.version).path("stram");
   // the filter should convert to the right version
-  ub = UriBuilder.fromUri("http://; + 
info.appMasterTrackingUrl).path(WebServices.PATH).path(WebServices.VERSION).path("stram");
+  String url;
+  if (!info.appMasterTrackingUrl.startsWith("http://;)
+  && !info.appMasterTrackingUrl.startsWith("https://;)) {
+url = "http://; + info.appMasterTrackingUrl;
--- End diff --

If it is a secure cluster, then fallback should be https?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #357: APEXCORE-488: Issues in SSL communication with ...

2016-07-11 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/357#discussion_r70355340
  
--- Diff: engine/src/main/java/com/datatorrent/stram/client/StramAgent.java 
---
@@ -204,7 +204,14 @@ private UriBuilder 
getStramWebURIBuilder(WebServicesClient webServicesClient, St
 if (info != null) {
   //ws = wsClient.resource("http://; + 
info.appMasterTrackingUrl).path(WebServices.PATH).path(info.version).path("stram");
   // the filter should convert to the right version
-  ub = UriBuilder.fromUri("http://; + 
info.appMasterTrackingUrl).path(WebServices.PATH).path(WebServices.VERSION).path("stram");
+  String url;
+  if (!info.appMasterTrackingUrl.startsWith("http://;)
+  && !info.appMasterTrackingUrl.startsWith("https://;)) {
+url = "http://; + info.appMasterTrackingUrl;
--- End diff --

It should be logged, saying falling back to http


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-07-09 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-core/pull/350

APEXCORE-222 Purging the stale data present in the bufferserver, direâ¦

â¦ctly from the streaming container.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-222

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #350


commit 36e3d333d0c4f22d07701cefacaacae0ab81a0eb
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-06-28T03:58:07Z

APEXCORE-222 purging of the buffer server is done from the streaming 
container, instead of StreamingContainerManager




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-07-09 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/350


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-08 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/298

[APEXMALHAR-2086] Kafka output operator: 0.9.0

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 3cf9c7ea9d2ba8d62586923d202fb5207c7dc45c
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka 0.9.0 output operators and unit tests.

1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
 Key in the Kafka message is used by the operator to track the tuples 
written by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-07-07 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r70022763
  
--- Diff: 
bufferserver/src/main/java/com/datatorrent/bufferserver/server/Server.java ---
@@ -199,6 +199,13 @@ private void handlePurgeRequest(PurgeRequestTuple 
request, final AbstractLengthP
 }
   }
 
+  public void purge(long windowId)
+  {
+for (DataList dataList: publisherBuffers.values()) {
--- End diff --

Purge inside the *Block* is not thread safe, but the Purge inside the 
*DataList* calls the Block purge using Synchronized block. So it should be fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-07 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/298

[APEXMALHAR-2086] Kafka output operator: 0.9.0

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 525ce5cca6536c88052e5e8bcc430b6acda06d57
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka 0.9.0 output operators and unit tests.

1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
 Key in the Kafka message is used by the operator to track the tuples 
written by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-07 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/298

[APEXMALHAR-2086] Kafka output operator: 0.9.0

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 525ce5cca6536c88052e5e8bcc430b6acda06d57
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka 0.9.0 output operators and unit tests.

1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
 Key in the Kafka message is used by the operator to track the tuples 
written by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-07 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-malhar/pull/298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-07 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/298

[APEXMALHAR-2086] Kafka output operator: 0.9.0

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 6bf316da76bf1d1871f52a114a307c05ff652376
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka 0.9.0 output operators and unit tests.

1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
 Key in the Kafka message is used by the operator to track the tuples 
written by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #324: Spillable Datastructures PR for review only

2016-07-05 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/324#discussion_r69600767
  
--- Diff: 
library/src/main/java/org/apache/apex/malhar/lib/state/spillable/TimeBasedPriorityQueue.java
 ---
@@ -0,0 +1,128 @@
+package org.apache.apex.malhar.lib.state.spillable;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+
+public class TimeBasedPriorityQueue
--- End diff --

Have you looked at 
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/DelayQueue.html ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-05 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-malhar/pull/298

[APEXMALHAR-2086] Kafka output operator: 0.9.0

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit 6bf316da76bf1d1871f52a114a307c05ff652376
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka 0.9.0 output operators and unit tests.

1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
 Key in the Kafka message is used by the operator to track the tuples 
written by it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output operator: 0.9....

2016-07-01 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/298#discussion_r69347651
  
--- Diff: 
kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java
 ---
@@ -0,0 +1,401 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.kafka;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.apex.malhar.lib.wal.FSWindowDataManager;
+import org.apache.apex.malhar.lib.wal.WindowDataManager;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.Callback;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator;
+
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Kafka output operator with exactly once processing semantics.
--- End diff --

@tweise & @siyuanh updated the JavaDoc. Please take a look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core issue #350: APEXCORE-222 Purging the stale data present in the buf...

2016-07-01 Thread sandeshh

Github user sandeshh commented on the issue:

https://github.com/apache/apex-core/pull/350
  
All the review comments has been addressed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output: 0.9.1 first c...

2016-06-28 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/298#discussion_r68862518
  
--- Diff: 
kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java
 ---
@@ -0,0 +1,369 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.kafka;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.apex.malhar.lib.wal.FSWindowDataManager;
+import org.apache.apex.malhar.lib.wal.WindowDataManager;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator;
+
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Kafka output operator with exactly once processing semantics under 
certain conditions.,
+ *
+ *  This operator uses *Key* to distinguish the messages written by 
particular instance of the Output operator.
+ *  Operator users can only use *value* for storing the data.
+ *
+ * @displayName Single Port Exactly Once Kafka Output(0.9.0)
+ * @category Messaging
+ * @tags output operator
+ *
+ */
+@org.apache.hadoop.classification.InterfaceStability.Evolving
+public class KafkaSinglePortExactlyOnceOutputOperator extends 
AbstractKafkaOutputOperator<String, T>
+implements Operator.CheckpointNotificationListener
+{
+  private transient String key;
+  private transient String appName;
+  private transient Integer operatorId;
+  private transient Long windowId;
+  private transient Map<T, Integer> partialWindowTuples = new HashMap<>();
+  private transient KafkaConsumer consumer;
+
+  private WindowDataManager windowDataManager = new FSWindowDataManager();
+  private final int KAFKA_CONNECT_ATTEMPT = 10;
+  private final String KEY_SEPARATOR = "#";
+  private final String KEY_SERIALIZER = 
"org.apache.kafka.common.serialization.StringDeserializer";
+  private final String VALUE_SERIALIZER = 
"org.apache.kafka.common.serialization.StringDeserializer";
+
+  public final transient DefaultInputPort inputPort = new 
DefaultInputPort()
+  {
+@Override
+public void process(T tuple)
+{
+  sendTuple(tuple);
+}
+  };
+
+  @Override
+  public void setup(Context.OperatorContext context)
+  {
+super.setup(context);
+
+this.operatorId = context.getId();
+this.windowDataManager.setup(context);
+this.appName = context.getValue(Context.DAGContext.APPLICATION_NAME);
+this.key = appName + KEY_SEPARATOR + (new Integer(operatorId));
+this.consumer = KafkaConsumerInit();
+  }
+
+  @Override
+  public void beginWindow(long windowId)
+  {
+this.windowId = windowId;
+
+if (windowId == windowDataManager.getLargestRecoveryWindow()) {
+  rebuildPartialWindow();
+}
+  }
+
+  @Override
+  public void checkpointed(long windowId)
+  {
+  }
+
+  @Override
+  public void committed(long windowId)
+  {
+

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output: 0.9.1 first c...

2016-06-28 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/298#discussion_r68862254
  
--- Diff: 
kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java
 ---
@@ -0,0 +1,369 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.kafka;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.apex.malhar.lib.wal.FSWindowDataManager;
+import org.apache.apex.malhar.lib.wal.WindowDataManager;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator;
+
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Kafka output operator with exactly once processing semantics under 
certain conditions.,
+ *
+ *  This operator uses *Key* to distinguish the messages written by 
particular instance of the Output operator.
+ *  Operator users can only use *value* for storing the data.
+ *
+ * @displayName Single Port Exactly Once Kafka Output(0.9.0)
+ * @category Messaging
+ * @tags output operator
+ *
+ */
+@org.apache.hadoop.classification.InterfaceStability.Evolving
+public class KafkaSinglePortExactlyOnceOutputOperator extends 
AbstractKafkaOutputOperator<String, T>
+implements Operator.CheckpointNotificationListener
+{
+  private transient String key;
+  private transient String appName;
+  private transient Integer operatorId;
+  private transient Long windowId;
+  private transient Map<T, Integer> partialWindowTuples = new HashMap<>();
+  private transient KafkaConsumer consumer;
+
+  private WindowDataManager windowDataManager = new FSWindowDataManager();
+  private final int KAFKA_CONNECT_ATTEMPT = 10;
+  private final String KEY_SEPARATOR = "#";
+  private final String KEY_SERIALIZER = 
"org.apache.kafka.common.serialization.StringDeserializer";
+  private final String VALUE_SERIALIZER = 
"org.apache.kafka.common.serialization.StringDeserializer";
+
+  public final transient DefaultInputPort inputPort = new 
DefaultInputPort()
+  {
+@Override
+public void process(T tuple)
+{
+  sendTuple(tuple);
+}
+  };
+
+  @Override
+  public void setup(Context.OperatorContext context)
+  {
+super.setup(context);
+
+this.operatorId = context.getId();
+this.windowDataManager.setup(context);
+this.appName = context.getValue(Context.DAGContext.APPLICATION_NAME);
+this.key = appName + KEY_SEPARATOR + (new Integer(operatorId));
+this.consumer = KafkaConsumerInit();
+  }
+
+  @Override
+  public void beginWindow(long windowId)
+  {
+this.windowId = windowId;
+
+if (windowId == windowDataManager.getLargestRecoveryWindow()) {
+  rebuildPartialWindow();
+}
+  }
+
+  @Override
+  public void checkpointed(long windowId)
+  {
+  }
+
+  @Override
+  public void committed(long windowId)
+  {
+

[GitHub] apex-malhar pull request #298: [APEXMALHAR-2086] Kafka output: 0.9.1 first c...

2016-06-28 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-malhar/pull/298#discussion_r68860997
  
--- Diff: 
kafka/src/main/java/org/apache/apex/malhar/kafka/KafkaSinglePortExactlyOnceOutputOperator.java
 ---
@@ -0,0 +1,369 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.kafka;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.ExecutionException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.apex.malhar.lib.wal.FSWindowDataManager;
+import org.apache.apex.malhar.lib.wal.WindowDataManager;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DefaultInputPort;
+import com.datatorrent.api.Operator;
+
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
+import static 
org.apache.kafka.clients.consumer.ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG;
+
+/**
+ * Kafka output operator with exactly once processing semantics under 
certain conditions.,
+ *
+ *  This operator uses *Key* to distinguish the messages written by 
particular instance of the Output operator.
+ *  Operator users can only use *value* for storing the data.
+ *
+ * @displayName Single Port Exactly Once Kafka Output(0.9.0)
+ * @category Messaging
+ * @tags output operator
+ *
+ */
+@org.apache.hadoop.classification.InterfaceStability.Evolving
+public class KafkaSinglePortExactlyOnceOutputOperator extends 
AbstractKafkaOutputOperator<String, T>
+implements Operator.CheckpointNotificationListener
+{
+  private transient String key;
+  private transient String appName;
+  private transient Integer operatorId;
+  private transient Long windowId;
+  private transient Map<T, Integer> partialWindowTuples = new HashMap<>();
+  private transient KafkaConsumer consumer;
+
+  private WindowDataManager windowDataManager = new FSWindowDataManager();
+  private final int KAFKA_CONNECT_ATTEMPT = 10;
+  private final String KEY_SEPARATOR = "#";
+  private final String KEY_SERIALIZER = 
"org.apache.kafka.common.serialization.StringDeserializer";
+  private final String VALUE_SERIALIZER = 
"org.apache.kafka.common.serialization.StringDeserializer";
+
+  public final transient DefaultInputPort inputPort = new 
DefaultInputPort()
+  {
+@Override
+public void process(T tuple)
+{
+  sendTuple(tuple);
+}
+  };
+
+  @Override
+  public void setup(Context.OperatorContext context)
+  {
+super.setup(context);
+
+this.operatorId = context.getId();
+this.windowDataManager.setup(context);
+this.appName = context.getValue(Context.DAGContext.APPLICATION_NAME);
+this.key = appName + KEY_SEPARATOR + (new Integer(operatorId));
+this.consumer = KafkaConsumerInit();
+  }
+
+  @Override
+  public void beginWindow(long windowId)
+  {
+this.windowId = windowId;
+
+if (windowId == windowDataManager.getLargestRecoveryWindow()) {
+  rebuildPartialWindow();
+}
+  }
+
+  @Override
+  public void checkpointed(long windowId)
+  {
+  }
+
+  @Override
+  public void committed(long windowId)
+  {
+

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-27 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r68643226
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java ---
@@ -769,7 +769,11 @@ public void 
processHeartbeatResponse(ContainerHeartbeatResponse rsp)
 }
 
 if (rsp.committedWindowId != lastCommittedWindowId) {
+
   lastCommittedWindowId = rsp.committedWindowId;
+
+  bufferServer.purge(lastCommittedWindowId);
--- End diff --

I have created a jira for the extra window. Will update this PR accordingly.
https://issues.apache.org/jira/browse/APEXCORE-479


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-25 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r68496137
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java ---
@@ -769,7 +769,11 @@ public void 
processHeartbeatResponse(ContainerHeartbeatResponse rsp)
 }
 
 if (rsp.committedWindowId != lastCommittedWindowId) {
+
   lastCommittedWindowId = rsp.committedWindowId;
+
+  bufferServer.purge(lastCommittedWindowId);
--- End diff --

Wrote a sample application and tested this change by killing the 
containers. There was no issue and the recovery started from the expected 
window id ( committed + 1 ). Still not sure why we have to keep the window data 
of the committed window. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-25 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r68493910
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java ---
@@ -769,7 +769,11 @@ public void 
processHeartbeatResponse(ContainerHeartbeatResponse rsp)
 }
 
 if (rsp.committedWindowId != lastCommittedWindowId) {
+
   lastCommittedWindowId = rsp.committedWindowId;
+
+  bufferServer.purge(lastCommittedWindowId);
--- End diff --

Working on that, here are the current finding.

1. Previous purge kept one extra window before the committed window. 
2. Couldn't find the history for that in GitHub. That change is present 
atleast from 2013.

Next steps

Will conduct few test scenario to see why we need a window before the 
committed window in the buffer server.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-25 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r68493801
  
--- Diff: 
bufferserver/src/main/java/com/datatorrent/bufferserver/internal/DataList.java 
---
@@ -176,21 +176,20 @@ public void reset()
 numberOfInMemBlockPermits.set(MAX_COUNT_OF_INMEM_BLOCKS - 1);
   }
 
-  public void purge(final int baseSeconds, final int windowId)
--- End diff --

That was the suggestion by @vrozov to remove the extra API. I don't have 
any strong opinion about them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-25 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r68493761
  
--- Diff: engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java 
---
@@ -72,7 +72,7 @@
   protected final StreamingContainerManager dnmgr;
   private final UmbilicalProtocolLocalImpl umbilical;
   private InetSocketAddress bufferServerAddress;
-  private boolean perContainerBufferServer;
+  private boolean perContainerBufferServer = true;
--- End diff --

Agreed. I already changed it locally, will be in next update.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-23 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r68184470
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java ---
@@ -769,7 +769,11 @@ public void 
processHeartbeatResponse(ContainerHeartbeatResponse rsp)
 }
 
 if (rsp.committedWindowId != lastCommittedWindowId) {
+
   lastCommittedWindowId = rsp.committedWindowId;
+
+  bufferServer.purge(lastCommittedWindowId);
--- End diff --

In localcluster buffer server is not started by default making that change 
passes the test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #349: APEXCORE-470 New API in DAG - setOperatorAttrib...

2016-06-15 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/349#discussion_r67238893
  
--- Diff: api/src/main/java/com/datatorrent/api/DAG.java ---
@@ -236,11 +236,22 @@
   public abstract  void setAttribute(Attribute key, T value);
 
   /**
-   * setAttribute.
+   *
+   * Use {@link #setOperatorAttribute} instead
*/
--- End diff --

Addressed all the review comments. Travis builds fails randomly. I just 
updated JavaDoc, so it shouldn't have updated the Travis.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-14 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r67065992
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java ---
@@ -769,7 +769,11 @@ public void 
processHeartbeatResponse(ContainerHeartbeatResponse rsp)
 }
 
 if (rsp.committedWindowId != lastCommittedWindowId) {
+
   lastCommittedWindowId = rsp.committedWindowId;
+
+  bufferServer.purge(lastCommittedWindowId);
--- End diff --

That loopback communication is for the tuple flow. This change is not 
dependent on that. I can still make this a loopback from the streaming 
container. 

  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-14 Thread sandeshh

Github user sandeshh commented on a diff in the pull request:

https://github.com/apache/apex-core/pull/350#discussion_r67062269
  
--- Diff: 
engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java ---
@@ -769,7 +769,11 @@ public void 
processHeartbeatResponse(ContainerHeartbeatResponse rsp)
 }
 
 if (rsp.committedWindowId != lastCommittedWindowId) {
+
   lastCommittedWindowId = rsp.committedWindowId;
+
+  bufferServer.purge(lastCommittedWindowId);
--- End diff --

I don't think we have any plans to run BufferServer in different 
process/container. There is Jira to replace the loopback  communication between 
BufferServer and the operators with a queue. So considering that points, I used 
this approach.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-14 Thread sandeshh

GitHub user sandeshh reopened a pull request:

https://github.com/apache/apex-core/pull/350

APEXCORE-222 Purging the stale data present in the bufferserver, direâ¦

â¦ctly from the streaming container.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-222

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #350


commit c41cd38cce8c9772aa82c37acad0b6ebc4bec0e0
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-06-14T06:33:39Z

APEXCORE-222 Purging the stale data present in the bufferserver, directly 
from the streaming container.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #350: APEXCORE-222 Purging the stale data present in ...

2016-06-14 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/apex-core/pull/350


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] apex-core pull request #349: APEXCORE-470

2016-06-07 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/apex-core/pull/349

APEXCORE-470 

1. Added the new API - setOperatorAttribute 
2. updated the unit tests,
3. marked "setAttribute(Operator..."  as deprecated

@PramodSSImmaneni please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/apex-core APEXCORE-470

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/349.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #349


commit bcd3426ba65f5291925dedb1e4fdfef4c80096ed
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-06-07T23:27:26Z

APEXCORE-470 - Added the new api setOperatorAttribute and updated the 
tests, also marked setAttribute(Operator... as deprecated




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-apex-malhar pull request: For review only [APEXMALHAR-...

2016-05-26 Thread sandeshh

GitHub user sandeshh opened a pull request:

https://github.com/apache/incubator-apex-malhar/pull/298

*For review only* [APEXMALHAR-2086] Kafka output: 0.9.1 first cut

Kafka output exactly once operator and the regular output operator.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sandeshh/incubator-apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #298


commit ea26306b1bc4a3d13ef3e28a222032ad7d1f6955
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

Kafka output: 0.9.1 first cut




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-apex-core pull request: Review only Prototype of runni...

2016-05-23 Thread sandeshh

Github user sandeshh closed the pull request at:

https://github.com/apache/incubator-apex-core/pull/340


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

78 matches

Mail list logo