[
https://issues.apache.org/jira/browse/STORM-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681376#comment-14681376
]
ASF GitHub Bot commented on STORM-954:
--------------------------------------
GitHub user arunmahadevan opened a pull request:
https://github.com/apache/storm/pull/673
[STORM-954] Topology event inspector
At a high level this enables users to optionally view the tuples (and some
metadata) generated at individual spouts and bolts in a topology. A few things
like the list of fields to log etc needs to be finalized, however raising a PR
to get the review started.
Users can enable the option globally or at spout/bolt level via the storm
UI. A new api is added to nimbus to accept global or component level debug flag
and pass on this info to the workers by updating `StormBase`. The workers
receive the updates via already existing timer (that currently refreshes the
component `active` status) and based on the flags, sends the tuples and
additional metadata to an internal `EventLogger` bolt.
The `EventLogger` is wired as an internal system bolt and outgoing streams
are added from all spouts and bolts to forward tuples to the event logger. The
event logging can be controlled via config option
`topology.eventlogger.executors` (similar to topology.acker.executors). The
forwarded tuples are field grouped on `component-id` and thus tuples from same
spout or bolt always ends up in the same EventLogger task (if there are
multiple executors defined) and the same logic is used to display the UI link
so that users can easily view the logs via the existing event logging feature.
This may be enhanced in future to load the tuples to a database and/or index
and provide search capabilities etc.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arunmahadevan/storm STORM-954-PR
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/673.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #673
----
commit a924e9efb461cd86cee7ff59d867c8cf2118f1e0
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-03T07:04:30Z
Event (tuple) logging and viewing support.
1. Added EventLogger internal bolt
2. Forward tuples + metadata from spouts/bolts to the EventLogger.
3. Added UI link in component summary to view the logged events.
commit 26b02706201e3dc92b1dff8398795ae78d6753b6
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-06T12:56:38Z
Nimbus debug api first cut
1. Added debug flag to storm base (currently boolean)
2. Added a new nimbus api for setting the flag
3. UI changes to enable and disable debugging at the topology level.
commit f8c4c5bf428c812ffa51eaabed0c8c1d04793cb4
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-07T05:45:42Z
Checking in the py files that were modifed by genthrift
commit d1e5ec98732e77409e6303b1ec3708a44e6a7bb9
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-07T08:58:14Z
Debug options support for components
commit 5914944f6ac7ad637a8727ab04fed0c1a5e2be1e
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-09T12:47:19Z
nimbus and stormbase changes for component level debug flag
commit 306ec57d9cebaaa8f5b9caa0c6fcfb256702bc77
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-09T12:57:47Z
nimbus api and stormbase changes to support component level debug
commit 7940d0c57c07c6b168617a0daaf182895cec8028
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-10T09:38:10Z
Merging upstream/master
commit 35990fa28aa660b389afb00ca3ecfe38fdf519db
Author: Arun Mahadevan <[email protected]>
Date: 2015-08-11T06:56:45Z
Fixed unit tests and a few issues
----
> Toplogy Event Inspector
> -----------------------
>
> Key: STORM-954
> URL: https://issues.apache.org/jira/browse/STORM-954
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Sriharsha Chintalapani
> Assignee: Arun Mahadevan
>
> •Ability to view tuples flowing through the topology
> •Ability to turn on/off debug events without having to stop/restart topology
> •Default debug events is off
> •User should be able to select a specific Spout or Bolt and see incoming
> events and outgoing events
> •We could put a configurable numbers of events to view (e.g. last 100 events
> or last 1 minute)
> •Tuple stream to have following info
> •Message id, batch/transaction id, name/value pair, timestamp, acked (boolean)
> •All the above to be available from Storm UI
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
