[jira] [Created] (STORM-2975) Worker died if KafkaSpout catched a kafka CommitFailedException
Xinyu Han created STORM-2975: Summary: Worker died if KafkaSpout catched a kafka CommitFailedException Key: STORM-2975 URL: https://issues.apache.org/jira/browse/STORM-2975 Project: Apache Storm Issue Type: Bug Components: storm-kafka-client Affects Versions: 1.1.1 Reporter: Xinyu Han Fix For: 1.x org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:792) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:738) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:808) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:204) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:167) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:127) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:488) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:348) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:184) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:605) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1173) ~[stormjar.jar:?] at org.apache.storm.kafka.spout.KafkaSpout.commitOffsetsForAckedTuples(KafkaSpout.java:384) ~[stormjar.jar:?] at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:220) ~[stormjar.jar:?] at org.apache.storm.daemon.executor$fn__4962$fn__4977$fn__5008.invoke(executor.clj:646) ~[storm-core-1.1.1.jar:1.1.1] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) [storm-core-1.1.1.jar:1.1.1] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2963) Updates to Performance.md
[ https://issues.apache.org/jira/browse/STORM-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376431#comment-16376431 ] Roshan Naik commented on STORM-2963: This is a catch all jira. As things come up I will keep accumulating them, and finally create a PR close to the 2.0 release. > Updates to Performance.md > -- > > Key: STORM-2963 > URL: https://issues.apache.org/jira/browse/STORM-2963 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2908) Document Metrics V2 for 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376375#comment-16376375 ] Jungtaek Lim commented on STORM-2908: - [~ptgoetz] Could you spend time to work on this issue? This is inconsistency between 1.x version line (after 1.2.0) and 2.0.0. > Document Metrics V2 for 2.0.0 > - > > Key: STORM-2908 > URL: https://issues.apache.org/jira/browse/STORM-2908 > Project: Apache Storm > Issue Type: Task > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: P. Taylor Goetz >Priority: Blocker > > This issue is to track follow-up task on Metrics V2: documentation. > The purpose for 'blocker' is to ensure this to be finished before releasing > Storm 2.0.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2909) New Metrics Reporting API - for 2.0.0
[
https://issues.apache.org/jira/browse/STORM-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376376#comment-16376376
]
Jungtaek Lim commented on STORM-2909:
-
[~ptgoetz]
Could you spend time to work on this issue? This is inconsistency between 1.x
version line (after 1.2.0) and 2.0.0.
> New Metrics Reporting API - for 2.0.0
> -
>
> Key: STORM-2909
> URL: https://issues.apache.org/jira/browse/STORM-2909
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client
>Affects Versions: 2.0.0
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>Priority: Blocker
>
> This is a proposal to provide a new metrics reporting API based on [Coda
> Hale's metrics library | http://metrics.dropwizard.io/3.1.0/] (AKA
> Dropwizard/Yammer metrics).
> h2. Background
> In a [discussion on the dev@ mailing list |
> http://mail-archives.apache.org/mod_mbox/storm-dev/201610.mbox/%3ccagx0urh85nfh0pbph11pmc1oof6htycjcxsxgwp2nnofukq...@mail.gmail.com%3e]
> a number of community and PMC members recommended replacing Storm’s metrics
> system with a new API as opposed to enhancing the existing metrics system.
> Some of the objections to the existing metrics API include:
> # Metrics are reported as an untyped Java object, making it very difficult to
> reason about how to report it (e.g. is it a gauge, a counter, etc.?)
> # It is difficult to determine if metrics coming into the consumer are
> pre-aggregated or not.
> # Storm’s metrics collection occurs through a specialized bolt, which in
> addition to potentially affecting system performance, complicates certain
> types of aggregation when the parallelism of that bolt is greater than one.
> In the discussion on the developer mailing list, there is growing consensus
> for replacing Storm’s metrics API with a new API based on Coda Hale’s metrics
> library. This approach has the following benefits:
> # Coda Hale’s metrics library is very stable, performant, well thought out,
> and widely adopted among open source projects (e.g. Kafka).
> # The metrics library provides many existing metric types: Meters, Gauges,
> Counters, Histograms, and more.
> # The library has a pluggable “reporter” API for publishing metrics to
> various systems, with existing implementations for: JMX, console, CSV, SLF4J,
> Graphite, Ganglia.
> # Reporters are straightforward to implement, and can be reused by any
> project that uses the metrics library (i.e. would have broader application
> outside of Storm)
> As noted earlier, the metrics library supports pluggable reporters for
> sending metrics data to other systems, and implementing a reporter is fairly
> straightforward (an example reporter implementation can be found here). For
> example if someone develops a reporter based on Coda Hale’s metrics, it could
> not only be used for pushing Storm metrics, but also for any system that used
> the metrics library, such as Kafka.
> h2. Scope of Effort
> The effort to implement a new metrics API for Storm can be broken down into
> the following development areas:
> # Implement API for Storms internal worker metrics: latencies, queue sizes,
> capacity, etc.
> # Implement API for user defined, topology-specific metrics (exposed via the
> {{org.apache.storm.task.TopologyContext}} class)
> # Implement API for storm daemons: nimbus, supervisor, etc.
> h2. Relationship to Existing Metrics
> This would be a new API that would not affect the existing metrics API. Upon
> completion, the old metrics API would presumably be deprecated, but kept in
> place for backward compatibility.
> Internally the current metrics API uses Storm bolts for the reporting
> mechanism. The proposed metrics API would not depend on any of Storm's
> messaging capabilities and instead use the [metrics library's built-in
> reporter mechanism |
> http://metrics.dropwizard.io/3.1.0/manual/core/#man-core-reporters]. This
> would allow users to use existing {{Reporter}} implementations which are not
> Storm-specific, and would simplify the process of collecting metrics.
> Compared to Storm's {{IMetricCollector}} interface, implementing a reporter
> for the metrics library is much more straightforward (an example can be found
> [here |
> https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/ConsoleReporter.java].
> The new metrics capability would not use or affect the ZooKeeper-based
> metrics used by Storm UI.
> h2. Relationship to JStorm Metrics
> [TBD]
> h2. Target Branches
> [TBD]
> h2. Performance Implications
> [TBD]
> h2. Metrics Namespaces
> [TBD]
> h2. Metrics Collected
> *Worker*
> || Namespace || Metric Type || Description ||
> *Nimbus*
> || Namespace || Metric Type || Description ||
> *Supervisor*
> || Names
[jira] [Closed] (STORM-1433) StormSQL Phase II
[ https://issues.apache.org/jira/browse/STORM-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim closed STORM-1433. --- Resolution: Fixed All subtasks completed. > StormSQL Phase II > - > > Key: STORM-1433 > URL: https://issues.apache.org/jira/browse/STORM-1433 > Project: Apache Storm > Issue Type: Epic > Components: storm-sql >Reporter: Haohui Mai >Priority: Major > > This epic tracks the effort of the phase II development of StormSQL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2963) Updates to Performance.md
[ https://issues.apache.org/jira/browse/STORM-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376365#comment-16376365 ] Jungtaek Lim commented on STORM-2963: - [~roshan_naik] Could you elaborate which updates you have in mind? From what it's issue summary and description, I can't see whether it is needed for Storm 2.0.0 or not. > Updates to Performance.md > -- > > Key: STORM-2963 > URL: https://issues.apache.org/jira/browse/STORM-2963 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2965) Interpret wildcard in classpath correctly when reading config from classpath
[ https://issues.apache.org/jira/browse/STORM-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2965. - Resolution: Fixed Fix Version/s: 2.0.0 Merged into master. > Interpret wildcard in classpath correctly when reading config from classpath > > > Key: STORM-2965 > URL: https://issues.apache.org/jira/browse/STORM-2965 > Project: Apache Storm > Issue Type: Bug > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 4h > Remaining Estimate: 0h > > While reading configuration file from classpath at > Utils.getConfigFromClasspath(), it doesn't interpret wildcard (\*) as Java > classpath does. It should behave same as Java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-2649) Update config validation check to give better information
[ https://issues.apache.org/jira/browse/STORM-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Hajós reassigned STORM-2649: Assignee: Gergely Hajós > Update config validation check to give better information > - > > Key: STORM-2649 > URL: https://issues.apache.org/jira/browse/STORM-2649 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Gergely Hajós >Priority: Major > Labels: newbie > > As part of submitting a topology we need to serialize the config as JSON. > The check right now is rather bad. it just calls Utils.isValidConf and throws > some generic Exception about problems. > https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/utils/Utils.java#L978-L980 > It would be much better to have a new function/method that would check if > they are equal and if not it would walk through them looking for which > configs are different when it finds one it includes which configs and how > they are different in the exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2971) Replace Flux storm-kafka example with an equivalent example for storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2971: -- Labels: pull-request-available (was: ) > Replace Flux storm-kafka example with an equivalent example for > storm-kafka-client > -- > > Key: STORM-2971 > URL: https://issues.apache.org/jira/browse/STORM-2971 > Project: Apache Storm > Issue Type: Sub-task > Components: Flux >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (STORM-2911) SpoutConfig is serializable but does not declare a serialVersionUID field
[ https://issues.apache.org/jira/browse/STORM-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Hajós reassigned STORM-2911: Assignee: Gergely Hajós > SpoutConfig is serializable but does not declare a serialVersionUID field > - > > Key: STORM-2911 > URL: https://issues.apache.org/jira/browse/STORM-2911 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 2.0.0, 1.1.1, 1.2.0, 1.0.5 >Reporter: Stig Rohde Døssing >Assignee: Gergely Hajós >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > See > https://stackoverflow.com/questions/48420085/kafkaspout-is-not-emmiting-message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2911) SpoutConfig is serializable but does not declare a serialVersionUID field
[ https://issues.apache.org/jira/browse/STORM-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376100#comment-16376100 ] Stig Rohde Døssing commented on STORM-2911: --- [~ghajos] I've added you to the contributors role, you should be able to assign the ticket now. Let me know if it doesn't work. > SpoutConfig is serializable but does not declare a serialVersionUID field > - > > Key: STORM-2911 > URL: https://issues.apache.org/jira/browse/STORM-2911 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 2.0.0, 1.1.1, 1.2.0, 1.0.5 >Reporter: Stig Rohde Døssing >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > See > https://stackoverflow.com/questions/48420085/kafkaspout-is-not-emmiting-message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2911) SpoutConfig is serializable but does not declare a serialVersionUID field
[ https://issues.apache.org/jira/browse/STORM-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16376086#comment-16376086 ] Gergely Hajós commented on STORM-2911: -- [~Srdo] Can you please give me access to assign this ticket myself? > SpoutConfig is serializable but does not declare a serialVersionUID field > - > > Key: STORM-2911 > URL: https://issues.apache.org/jira/browse/STORM-2911 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 2.0.0, 1.1.1, 1.2.0, 1.0.5 >Reporter: Stig Rohde Døssing >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > See > https://stackoverflow.com/questions/48420085/kafkaspout-is-not-emmiting-message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2911) SpoutConfig is serializable but does not declare a serialVersionUID field
[ https://issues.apache.org/jira/browse/STORM-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2911: -- Labels: newbie pull-request-available (was: newbie) > SpoutConfig is serializable but does not declare a serialVersionUID field > - > > Key: STORM-2911 > URL: https://issues.apache.org/jira/browse/STORM-2911 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 2.0.0, 1.1.1, 1.2.0, 1.0.5 >Reporter: Stig Rohde Døssing >Priority: Major > Labels: newbie, pull-request-available > > See > https://stackoverflow.com/questions/48420085/kafkaspout-is-not-emmiting-message -- This message was sent by Atlassian JIRA (v7.6.3#76005)
