[GitHub] samza pull request #519: SAMZA-1712: Tear down ZookeeperServer connections i...

[shanthoosh]

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/519

SAMZA-1712: Tear down ZookeeperServer connections in ZkClient on interrupts.

**Problem:** 

If a thread executing zkClient.close is interrupted, currently we swallow 
the ZkInterruptedException and proceed without closing the zookeeper 
connection. 

This leads to ephemeral nodes of StreamProcessor lurking around in 
zookeeper after StreamProcessor shutdown.

Users had to wait till zookeeper server session timeout for the ephemeral 
nodes to get deleted. 

**Change:**

Add fixed retries on InterruptedException when closing the zkClient.

Misc changes:
* Remove unnecessary null checks.
* Remove unnecessary typecasts.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza 
handle_interrupted_exception_in_zkclient_close

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #519


commit 42543da56701d0476e731860b0157a8af4365f56
Author: Shanthoosh Venkataraman 
Date:   2018-05-10T02:32:24Z

Add fixed retries on InterruptedException when closing the zkClient.

Misc changes:
* Remove unnecessary null checks.
* Remove unnecessary typecasts.




---

[GitHub] samza pull request #518: Snapshot reporter flush during shutdown, exception ...

[rayman7718]

GitHub user rayman7718 opened a pull request:

https://github.com/apache/samza/pull/518

Snapshot reporter flush during shutdown, exception metric addition

…pshot reporter.

Switched producer-shutdown logic to ensure metric-flush at shutdown-time

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rayman7718/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #518


commit 10ef7ad64a32d212da6dc2d4a331b49ba857675f
Author: Ray Matharu 
Date:   2018-05-10T23:31:07Z

Added exception metric (as string-guage), to be emitted using the snapshot 
reporter.
Switched producer-shutdown logic to ensure metric-flush at shutdown-time




---

[GitHub] samza pull request #517: SAMZA-1559 moving hello-samza repo to samza repo an...

[fredji97]

GitHub user fredji97 opened a pull request:

https://github.com/apache/samza/pull/517

SAMZA-1559 moving hello-samza repo to samza repo and put in samza-samples 
directory

Tested and jobs can be deployed in the new place. 
After this is committed, we will do the following two in order: 
1. adjust the documentation to use the sample jobs from the new place;
2. remove the old hello-samza repo. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fredji97/samza master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/517.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #517


commit 851f3db9b0afcb11eca72808040750ce185cf7f6
Author: Fred Ji 
Date:   2018-05-10T21:42:51Z

SAMZA-1559 moving hello-samza repo to samza repo and put in samza-samples 
directory




---

[VOTE] Apache Samza 0.14.1 RC1

[Xinyu Liu]

Hi, All,

This is a call for a vote on a release of Apache Samza 0.14.1. Thanks to
everyone who has contributed to this release.

The release candidate can be downloaded from here:
http://home.apache.org/~xinyu/samza-0.14.1-rc1/

The release candidate is signed with pgp key C31D7061, which can be found
on keyservers: http://pgp.mit.edu/pks/lookup?op=get=0xC31D7061

The git tag is release-0.14.1-rc0 and signed with the same pgp key:
https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=refs/tags/release-0.14.1-rc1

Test binaries have been published to Maven's staging repository, and are
available here:
https://repository.apache.org/content/repositories/orgapachesamza-1047/

46 issues were resolved for this release: https://issues.apache.org/
jira/projects/SAMZA/versions/12343155

The vote will be open for 72 hours (ending at 3:00PM Tuesday, 05/15/2018).

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

Thanks,
Xinyu

[GitHub] samza pull request #516: Remove the iterable interface from KeyValueSnapshot

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/516


---

Re: [VOTE] SEP-12: Integration Test Framework

[Sanil Jain]

Hi All,


This is a call for a vote for Samza's Integration Test Framework as described 
by:

https://cwiki.apache.org/confluence/display/SAMZA/SEP-12%3A+Integration+Test+Framework

The vote will be open for 5 days (ending at 2:00PM Thursday, 05/17/2018).

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

Link to the discuss mailing thread:


http://mail-archives.apache.org/mod_mbox/samza-dev/201805.mbox/%3CDM5PR21MB02827A6FA9F47CB8EF99A339A2810%40DM5PR21MB0282.namprd21.prod.outlook.com%3E


Please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

Thanks
Sanil


From: Sanil Jain 
Sent: Thursday, May 10, 2018 1:59:59 PM
To: dev@samza.apache.org
Subject: [VOTE] SEP-12: Integration Test Framework

Hi All,


This is a call for a vote for Samza's Integration Test Framework as described 
by:


https://cwiki.apache.org/confluence/display/SAMZA/SEP-12%3A+Integration+Test+Framework


The vote will be open for 5 days (ending at 2:00PM Thursday, 05/17/2018).

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

Thanks
Sanil

[VOTE] SEP-12: Integration Test Framework

[Sanil Jain]

Hi All,


This is a call for a vote for Samza's Integration Test Framework as described 
by:


https://cwiki.apache.org/confluence/display/SAMZA/SEP-12%3A+Integration+Test+Framework


The vote will be open for 5 days (ending at 2:00PM Thursday, 05/17/2018).

Please download the release candidate, check the hashes/signature, build it
and test it, and then please vote:

[ ] +1 approve

[ ] +0 no opinion

[ ] -1 disapprove (and reason why)

Thanks
Sanil

[GitHub] samza pull request #516: Remove the iterable interface from KeyValueSnapshot

[xinyuiscool]

GitHub user xinyuiscool opened a pull request:

https://github.com/apache/samza/pull/516

Remove the iterable interface from KeyValueSnapshot

The iterable interface makes it hard for the users to close it after using.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xinyuiscool/samza kv-snapshot

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/516.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #516


commit 3584a2158e8a1d0635780427bfb9db6f7184d2f4
Author: xinyuiscool 
Date:   2018-05-10T20:24:30Z

Remove the iterable interface from KeyValueSnapshot




---

[GitHub] samza pull request #514: Fixed test failure for TestRocksDbKeyValueStoreJava...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/514


---

[GitHub] samza pull request #509: SAMZA-1700 : Clean up SystemAdmins instance creatio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/509


---

[GitHub] samza pull request #515: SAMZA-1711: Re-enable existing standalone integrati...

[shanthoosh]

GitHub user shanthoosh opened a pull request:

https://github.com/apache/samza/pull/515

SAMZA-1711: Re-enable existing standalone integration tests.

**Changes:**

* Enable all existing standalone integration tests except 
`TestZkStreamProcessorSession`(`TestZkStreamProcessorSession` is flaky. It 
spawns `x` StreamProcessors and kills one StreamProcessor through zookeeper 
session expiration. Sleeps for 5 seconds and proceeds to do validation. If the 
rebalancing phase takes longer the sleep time, validation fails).
* Remove zookeeper unavailable unit test from LocalApplicationRunner(Race 
condition in zookeeper shutdown fails other tests). The deleted test will be 
added back in a separate test class.
* Increase zookeeper server minimum session timeout from 6 seconds to 120 
seconds.
* Add assertions to validate if kafka topics setup were successful before 
the unit tests.

**Validation:** 

Verified by running the following script on top of this patch.

```bash
i=0
while [ $i -lt 50 ]; do
i=`expr $i + 1`
echo "Run " +$i 
./gradlew clean :samza-test:test 
-Dtest.single="TestZkLocalApplicationRunner" --debug --stacktrace >> 
~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestZkStreamProcessor" 
--debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test -Dtest.single="TestStreamProcessor" 
--debug --stacktrace >> ~/test-logs_10
./gradlew clean :samza-test:test 
-Dtest.single="TestZkStreamProcessorFailures" --debug --stacktrace >> 
~/test-logs_10
done;
```


**Result:**

```bash
[svenkata@svenkata-ld2 samza]$ grep 'BUILD SUCCESS' ~/test-logs_10  | wc -l 
200
[svenkata@svenkata-ld2 samza]$ grep 'BUILD FAIL' ~/test-logs_10  | wc -l 
0
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shanthoosh/samza turn_all_integration_tests_on

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #515


commit 549efb7876fbad9ca3789eb68cdb73e495627b4f
Author: Shanthoosh Venkataraman 
Date:   2018-04-30T04:28:14Z

SAMZA-1692: Standalone stability fixes.

- Currently, on session expiration processorListener with incorrect 
generationId is registered with zookeeper(ZkUtils generationId is incremented 
on reconnect but the generationId in processorListener is zero all the time). 
When a session reconnect happens to a processor successive to leader, leader 
expiration event will be skipped. This will prevent leader re-election on a 
current leader death and will stall the processors group. Fix is to 
reinstantiate and then register processorChangeListener on session expiration.
- Add processorId to debounce thread name (this can aid debugging when 
multiple processors are running within a jvm).
- After ScheduleAfterDebounceTime queue is shutdown, don't accept new 
schedule requests. Current ZkJobCoordinator shutdown sequence comprise of the 
following steps
 - Shutdown the ScheduleAfterDebounceTime queue.
 - Stop the zkClient  and relinquish it's resources.

After we shutdown ScheduleAfterDebounceTime and before zkclient is stopped, 
any new operations can be scheduled in ScheduleAfterDebounceTime queue. This 
will result in RejectedExecutionException, since executorService is stopped.

```
Caused by: java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@23f962a8 
rejected from java.util.concurrent.ScheduledThreadPoolExecutor@43408be8
```

commit 6a07a960bbb2baf5d3149d0384ba5e98ecfd82fc
Author: Shanthoosh Venkataraman 
Date:   2018-05-07T20:23:58Z

Review comments.

commit 5ea977cf7b469653c2a588abeb3f46c344cfccfc
Author: Shanthoosh Venkataraman 
Date:   2018-05-09T23:30:48Z

Reenable all disabled standalone integration tests.




---

[GitHub] samza pull request #514: Fixed test failure for TestRocksDbKeyValueStoreJava...

[prateekm]

GitHub user prateekm opened a pull request:

https://github.com/apache/samza/pull/514

Fixed test failure for TestRocksDbKeyValueStoreJava#testPerf

Iterators (incl. those obtained from snapshots) must be closed before store 
close.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/prateekm/samza rocksdb-test-fi

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/samza/pull/514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #514


commit 93fb295e684aa7cb2571cad663ba38377cd165ed
Author: Prateek Maheshwari 
Date:   2018-05-10T02:15:08Z

Fixed test failure for TestRocksDbKeyValueStoreJava#testIterate

commit be117930dd4b91803c286772d8491fe6e281a6a0
Author: Prateek Maheshwari 
Date:   2018-05-10T17:33:02Z

Fixed test failure for TestRocksDbKeyValueStoreJava#testPerf




---

[CANCEL][VOTE] Apache Samza 0.14.1 RC0

[Xinyu Liu]

We need an performance fix for eventhub system producer (SAMZA-1706) as
well as a fix for the unit test in TestRocksDbKeyValueStoreJava. Cancel
this vote and will submit another RC soon.

Thanks,
Xinyu

Re: [VOTE] Apache Samza 0.14.1 RC0

[Xinyu Liu]

We are pulling in a fix in SAMZA-1706: lazy initialization for eventhub
system producer, and also a fix for test failure in
TestRocksDbKeyValueStoreJava#testIterate in this release. RC0 is cancelled.
Will send out the cancellation email shortly.

Thanks,
Xinyu

On Tue, May 8, 2018 at 6:48 PM, Xinyu Liu  wrote:

> Hi, All,
>
> This is a call for a vote on a release of Apache Samza 0.14.1. Thanks to
> everyone who has contributed to this release.
>
> The release candidate can be downloaded from here:
> http://home.apache.org/~xinyu/samza-0.14.1-rc0/
>
> The release candidate is signed with pgp key C31D7061, which can be found
> on keyservers: http://pgp.mit.edu/pks/lookup?op=get=0xC31D7061
>
> The git tag is release-0.14.1-rc0 and signed with the same pgp key:
> https://git-wip-us.apache.org/repos/asf?p=samza.git;a=tag;h=
> refs/tags/release-0.14.1-rc0
>
> Test binaries have been published to Maven's staging repository, and are
> available here: https://repository.apache.org/content/repositories/
> orgapachesamza-1046/
>
> 46 issues were resolved for this release: https://issues.apache.org/
> jira/projects/SAMZA/versions/12343155
>
> The vote will be open for 72 hours (ending at 7:00PM Friday, 05/11/2018).
>
> Please download the release candidate, check the hashes/signature, build
> it and test it, and then please vote:
>
> [ ] +1 approve
>
> [ ] +0 no opinion
>
> [ ] -1 disapprove (and reason why)
>
> Thanks,
> Xinyu
>

[GitHub] samza pull request #513: Fixed test failure for TestRocksDbKeyValueStoreJava...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/samza/pull/513


---