GitHub user nssalian reopened a pull request:
https://github.com/apache/spark/pull/7250
[SPARK-8743] [Streaming]: Deregister Codahale metrics for streaming when
StreamingContext is closed
The issue link: https://issues.apache.org/jira/browse/SPARK-8743
Deregister Codahale metrics for streaming when StreamingContext is closed
Design:
Adding the method calls in the appropriate start() and stop () methods for
the StreamingContext
Actions in the PullRequest:
1) Added the registerSource method call to the start method for the
Streaming Context.
2) Added the removeSource method to the stop method.
3) Added comments for both 1 and 2 and comment to show initialization of
the StreamingSource
Requesting Review.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nssalian/spark SPARK-8743
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7250.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7250
----
commit 92fa04b16cad1e945ab8a4d1be752f08a241e922
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-07T03:16:51Z
SPARK-8743: Added the registerSource method call to the start method for
the Streaming Context. Added the removeSource method to the stop method. Added
comments for both
commit a665965a40c7b49cc13d8ab38f8da194693d9845
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-07T07:23:20Z
Added // instead of /** for commenting in code
commit 7621adf368112056eb2e137f62adc429851fa570
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-07T10:36:35Z
Added indentation and Space at the comment on line 578; Registering..
commit 18bcc7e164b46ecd969a266579f4349444373a0c
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-07T23:20:29Z
Added test case for de-register metrics and made a change to the scope of
the sources ArrayBuffer
commit e4f00d7577dec22278f5b6243791c4c33e0eb373
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-08T18:56:30Z
Added additional variable to check the updated Sources size value to
compare with the original size after removal
commit f5e47e0725adee6792081a8a4ed765ede1759e9c
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-09T01:02:33Z
Added the removeSource method in try
commit d04fd2a2b2e4433d49c0860c4d4028564081db31
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-09T16:23:17Z
Removed the assert for the env field, added the registerSource line in the
INITIALIZED block and kept the removeSource() in the ACTIVE block
commit e2c3bf82c226e38282a9a17b80771b58dcc6cc55
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-09T22:02:50Z
Added test to check registering and de-registering of streamingSource
commit 742398c334c71bcd1b2b702a9abc5e4ab1288d9e
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-09T22:08:55Z
Removed unused imports
commit ca081fa3effd0303d04a034af5c5a0e8facd3b2d
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-10T02:33:38Z
Moved the registerSource() call before line 601
commit 33a2091a4984b8e29143ab2e1202751a87e838b3
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-10T21:31:53Z
Changed scope of sources and corrected comments for helper
commit a67918cb9732936ea84427f728086985f7319e3a
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-10T21:41:16Z
Removed extra line in Helper Methods section
commit 74598cec17a6ec54a64f3fc0f8c336d6ba19cc1e
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-11T02:16:18Z
Added helper method for private methods and changed the test logic to check
for Sources containing or not containing StreamingSource
commit e37a2f3cc3364f6819205b5eac39d0603eb91ac5
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T14:38:09Z
Changed import statements to remove unnecessary imports and add specific
imports
commit f54afcf78819ad30a59318164515457b47c31d7d
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T14:43:29Z
Removed types for fields in test for registering and deregistering metrics
commit ea0dc1a74848af8682bb8fc1e0f03b5261591f6e
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T22:38:00Z
Changed imports statements, negated test statement and removed postfix
commit a0f1950937c36d7f568c5be545cf72ba5afe36ee
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T22:43:39Z
Removed added comment to Assert for INITIALIZED state
commit 2a812878643a1e97e0113c7be7a195ae3c740b48
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T22:49:13Z
Removing the INITIALIZED check since after start() the state moves to
ACTIVE and this check fails
commit 5d3af311abe18e0db8cecb36141432af07d3afcb
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T23:03:34Z
Move the INITIALIZED state check to when the ssc is initialized
commit 299a57d0b909b2be968f17723736c66c0e61fdcd
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-07T03:16:51Z
SPARK-8743: Added the registerSource method call to the start method for
the Streaming Context. Added the removeSource method to the stop method. Added
comments for both
Added // instead of /** for commenting in code
Added indentation and Space at the comment on line 578; Registering..
Added test case for de-register metrics and made a change to the scope of
the sources ArrayBuffer
Added additional variable to check the updated Sources size value to
compare with the original size after removal
Added the removeSource method in try
Removed the assert for the env field, added the registerSource line in the
INITIALIZED block and kept the removeSource() in the ACTIVE block
Added test to check registering and de-registering of streamingSource
Removed unused imports
Moved the registerSource() call before line 601
Changed scope of sources and corrected comments for helper
Removed extra line in Helper Methods section
Added helper method for private methods and changed the test logic to check
for Sources containing or not containing StreamingSource
Changed import statements to remove unnecessary imports and add specific
imports
Removed types for fields in test for registering and deregistering metrics
Changed imports statements, negated test statement and removed postfix
Removed added comment to Assert for INITIALIZED state
Removing the INITIALIZED check since after start() the state moves to
ACTIVE and this check fails
Move the INITIALIZED state check to when the ssc is initialized
commit b86e80b753d263a753a3ee0b5f4744932bcd6015
Author: Neelesh Srinivas Salian <[email protected]>
Date: 2015-07-12T23:43:39Z
Merge branch 'SPARK-8743' of https://github.com/nssalian/spark into
SPARK-8743
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
