[jira] [Updated] (STORM-2306) Redeisgn Messaging Subsystem and switch to JCTools Queues
[ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2306: --- Description: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. was: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit#heading=h.59lnwp28s0q6 This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. > Redeisgn Messaging Subsystem and switch to JCTools Queues > - > > Key: STORM-2306 > URL: https://issues.apache.org/jira/browse/STORM-2306 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > > Details in these documents: > 1) *Redesign of the messaging subsystem* > https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing > This doc discusses the new design for the messaging system. Plus some of the > optimizations being made. > 2) *Choosing a high performance messaging queue:* > https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing > This doc looks into how fast hardware can do inter-thread messaging and why > we chose the JCTools queues. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2306) Redeisgn Messaging Subsystem and switch to JCTools Queues
[ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2306: --- Description: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit#heading=h.59lnwp28s0q6 This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. was: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit#heading=h.59lnwp28s0q6 This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can we can inter-thread messaging and why we chose the JCTools queues. > Redeisgn Messaging Subsystem and switch to JCTools Queues > - > > Key: STORM-2306 > URL: https://issues.apache.org/jira/browse/STORM-2306 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > > Details in these documents: > 1) *Redesign of the messaging subsystem* > https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit#heading=h.59lnwp28s0q6 > This doc discusses the new design for the messaging system. Plus some of the > optimizations being made. > 2) *Choosing a high performance messaging queue:* > https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing > This doc looks into how fast hardware can do inter-thread messaging and why > we chose the JCTools queues. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2657) Update SECURITY.MD
liuzhaokun created STORM-2657: - Summary: Update SECURITY.MD Key: STORM-2657 URL: https://issues.apache.org/jira/browse/STORM-2657 Project: Apache Storm Issue Type: Improvement Components: documentation Reporter: liuzhaokun Assignee: liuzhaokun There are several PRs about storm acl merged into master,so I update SECURITY.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2656) Cannot start Nimbus
David Bai created STORM-2656: Summary: Cannot start Nimbus Key: STORM-2656 URL: https://issues.apache.org/jira/browse/STORM-2656 Project: Apache Storm Issue Type: Question Reporter: David Bai After installation of apache storm, when run "sudo bin/storm nimbus", get error logs below. Already googled, if the problem is related to clean states, could you please tell me how to do that? Thanks! 2017-07-24 17:45:29.841 o.a.s.d.common main [INFO] Started statistics report plugin... 2017-07-24 17:45:29.871 o.a.s.d.nimbus main [INFO] Starting nimbus server for storm version '1.1.0' 2017-07-24 17:45:30.739 o.a.s.d.nimbus timer [ERROR] Error when processing event java.lang.RuntimeException: java.lang.RuntimeException: java.util.zip.ZipException: Not in GZIP format at org.apache.storm.serialization.GzipThriftSerializationDelegate.deserialize(GzipThriftSerializationDelegate.java:53) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.utils.Utils.deserialize(Utils.java:216) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.cluster$maybe_deserialize.invoke(cluster.clj:224) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.cluster$mk_storm_cluster_state$reify__4395.supervisor_info(cluster.clj:410) ~[storm-core-1.1.0.jar:1.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?] at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?] at org.apache.storm.daemon.nimbus$all_supervisor_info$fn__9895.invoke(nimbus.clj:448) ~[storm-core-1.1.0.jar:1.1.0] at clojure.core$map$fn__4553.invoke(core.clj:2624) ~[clojure-1.7.0.jar:?] at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?] at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?] at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?] at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?] at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?] at clojure.core$mapcat.doInvoke(core.clj:2660) ~[clojure-1.7.0.jar:?] at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?] at org.apache.storm.daemon.nimbus$all_supervisor_info.invoke(nimbus.clj:446) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.daemon.nimbus$all_supervisor_info.invoke(nimbus.clj:442) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.daemon.nimbus$read_all_supervisor_details.invoke(nimbus.clj:725) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.daemon.nimbus$compute_new_scheduler_assignments.invoke(nimbus.clj:858) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:986) ~[storm-core-1.1.0.jar:1.1.0] at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.7.0.jar:?] at org.apache.storm.daemon.nimbus$fn__11005$exec_fn__1364__auto11006$fn__11017.invoke(nimbus.clj:2446) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.timer$schedule_recurring$this__1737.invoke(timer.clj:105) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.timer$mk_timer$fn__1720$fn__1721.invoke(timer.clj:50) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.timer$mk_timer$fn__1720.invoke(timer.clj:42) ~[storm-core-1.1.0.jar:1.1.0] at clojure.lang.AFn.run(AFn.java:22) ~[clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.lang.RuntimeException: java.util.zip.ZipException: Not in GZIP format at org.apache.storm.utils.Utils.gunzip(Utils.java:288) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.serialization.GzipThriftSerializationDelegate.deserialize(GzipThriftSerializationDelegate.java:50) ~[storm-core-1.1.0.jar:1.1.0] ... 30 more Caused by: java.util.zip.ZipException: Not in GZIP format at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165) ~[?:1.8.0_131] at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) ~[?:1.8.0_131] at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) ~[?:1.8.0_131] at org.apache.storm.utils.Utils.gunzip(Utils.java:278) ~[storm-core-1.1.0.jar:1.1.0] at org.apache.storm.serialization.GzipThriftSerializationDelegate.deserialize(GzipThriftSerializationDelegate.java:50) ~[storm-core-1.1.0.jar:1.1.0] ... 30 more 2017-07-24 17:45:30.755 o.a.s.util timer
[jira] [Updated] (STORM-2655) Log user cannot view worker.log on Storm UI on secure cluster
[
https://issues.apache.org/jira/browse/STORM-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Li updated STORM-2655:
Priority: Major (was: Minor)
> Log user cannot view worker.log on Storm UI on secure cluster
> -
>
> Key: STORM-2655
> URL: https://issues.apache.org/jira/browse/STORM-2655
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-webapp
>Reporter: Ethan Li
>Assignee: Ethan Li
> Attachments: logusers.png, not-authorized.png
>
>
> "ethan" is a logs.users to a topology. But it is not able to view worker.log
> on Storm UI.
> The problem is that the [isAuthorizedLogUser
> function|https://github.com/apache/storm/blob/3c53dd516810824593855060a69c5e82d08323e7/storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/utils/ResourceAuthorizer.java#L74-L95]
> doesn't translate principal("et...@derekd.ystorm.net") to username("ethan")
> so that the authentication checking failed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (STORM-2655) Log user cannot view worker.log on Storm UI on secure cluster
Ethan Li created STORM-2655:
---
Summary: Log user cannot view worker.log on Storm UI on secure
cluster
Key: STORM-2655
URL: https://issues.apache.org/jira/browse/STORM-2655
Project: Apache Storm
Issue Type: Bug
Components: storm-webapp
Reporter: Ethan Li
Assignee: Ethan Li
Priority: Minor
Attachments: logusers.png, not-authorized.png
"ethan" is a logs.users to a topology. But it is not able to view worker.log on
Storm UI.
The problem is that the [isAuthorizedLogUser
function|https://github.com/apache/storm/blob/3c53dd516810824593855060a69c5e82d08323e7/storm-webapp/src/main/java/org/apache/storm/daemon/logviewer/utils/ResourceAuthorizer.java#L74-L95]
doesn't translate principal("et...@derekd.ystorm.net") to username("ethan") so
that the authentication checking failed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (STORM-2654) Error column in Storm UI loses original formatting
Ethan Li created STORM-2654: --- Summary: Error column in Storm UI loses original formatting Key: STORM-2654 URL: https://issues.apache.org/jira/browse/STORM-2654 Project: Apache Storm Issue Type: Improvement Reporter: Ethan Li Assignee: Ethan Li Priority: Trivial Attachments: unformated.png Whenever an exception stack trace is shown in the Storm UI in the error column, newlines are not being rendered. This makes it hard to read long stacktraces. Is it possible to preserve the formatting of the stack trace in this view? ---Reported by Pramod Biligiri Subba Rao @ Yahoo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-1280) port backtype.storm.daemon.logviewer to java
[ https://issues.apache.org/jira/browse/STORM-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1280. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~kabhwan], I merged this into master. > port backtype.storm.daemon.logviewer to java > > > Key: STORM-1280 > URL: https://issues.apache.org/jira/browse/STORM-1280 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Jungtaek Lim > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > This is providing a UI for accessing and searching logs. hiccup will need to > be replaced, possibly with just hard coded HTML + escaping. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2626) Provided a template for drpc-auth-acl.yaml
[ https://issues.apache.org/jira/browse/STORM-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2626. Resolution: Fixed Fix Version/s: 2.0.0 > Provided a template for drpc-auth-acl.yaml > -- > > Key: STORM-2626 > URL: https://issues.apache.org/jira/browse/STORM-2626 > Project: Apache Storm > Issue Type: Improvement >Reporter: liuzhaokun >Assignee: liuzhaokun > Fix For: 2.0.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > The default value of drpc.authorizer.acl.filename in defaults.yaml is > "drpc-auth-acl.yaml",so I think we should provided a template for > drpc-auth-acl.yaml. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-1347) ui changes to display the topology version.
[ https://issues.apache.org/jira/browse/STORM-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098617#comment-16098617 ] Ethan Li commented on STORM-1347: - Hi [~parth.brahmbhatt] We have a similar request for this feature. It seems that you haven't touched this issue for almost one year. Is that OK for me to take it over? Thanks very much! > ui changes to display the topology version. > > > Key: STORM-1347 > URL: https://issues.apache.org/jira/browse/STORM-1347 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2653) PaceMaker code improvement
[ https://issues.apache.org/jira/browse/STORM-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Li updated STORM-2653: Description: Some minor improvements in PaceMakerClient source code 1. add shutdown checking before connecting 2. add try-catch block for robustness. > PaceMaker code improvement > -- > > Key: STORM-2653 > URL: https://issues.apache.org/jira/browse/STORM-2653 > Project: Apache Storm > Issue Type: Improvement >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > Some minor improvements in PaceMakerClient source code > 1. add shutdown checking before connecting > 2. add try-catch block for robustness. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2646) NimbusClient Class cast exception when nimbus seeds is not an array of hosts
[ https://issues.apache.org/jira/browse/STORM-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098516#comment-16098516 ] Eugeniu Cararus commented on STORM-2646: Fix has been committed on: https://github.com/apache/storm/pull/2228/commits/732d0c276d4ee5c7dbf2422a5b5ed78baef5e838 should be merged to master > NimbusClient Class cast exception when nimbus seeds is not an array of hosts > > > Key: STORM-2646 > URL: https://issues.apache.org/jira/browse/STORM-2646 > Project: Apache Storm > Issue Type: Bug > Components: storm-client >Affects Versions: 1.0.3 >Reporter: Eugeniu Cararus > Labels: client, config, nimbus > Fix For: 2.0.0 > > Original Estimate: 2h > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2653) PaceMaker code improvement
Ethan Li created STORM-2653: --- Summary: PaceMaker code improvement Key: STORM-2653 URL: https://issues.apache.org/jira/browse/STORM-2653 Project: Apache Storm Issue Type: Improvement Reporter: Ethan Li Assignee: Ethan Li Priority: Trivial -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2614) Enhance stateful windowing to persist the window state
[ https://issues.apache.org/jira/browse/STORM-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098049#comment-16098049 ] Arun Mahadevan commented on STORM-2614: --- High level design: This builds on top of the existing state checkpointing mechanism (documented here - https://github.com/apache/storm/blob/master/docs/State-checkpointing.md). Theres nothing extra added to the underlying checkpointing mechanism itself and its pretty straightforward. The tuples in window (think a FIFO queue) are split into multiple partitions so that they are more manageable and can be distributed/sharded via the underlying key-value state (redis/hbase etc). The modified partitions are saved during a checkpoint. During iteration the partition are loaded on demand from the underlying state backend as they are accessed. A subset of the partitions that are most likely to be used again are cached in memory. During checkpoint, the following are saved : 1. Any modified or newly created window partitions. 2. Any state needed to recover the Trigger/Eviction policies. 3. State thats exposed to the user where the user may have saved some values. Since the KV state does not guarantee any specific ordering of the keys during iteration, a separate structure is maintained to store the ordered partition Ids which is used during iteration to retrieve the partitions in order. This is also saved during the checkpoint. The above mechanism kicks in only if user choses to use the windowed state persistence, otherwise the current behavior (keeping the tuples in an in-memory queue) is retained. > Enhance stateful windowing to persist the window state > -- > > Key: STORM-2614 > URL: https://issues.apache.org/jira/browse/STORM-2614 > Project: Apache Storm > Issue Type: Bug >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan > Time Spent: 1.5h > Remaining Estimate: 0h > > Right now the tuples in window are stored in memory. This limits the usage to > windows that fit in memory and the source tuples cannot be acked until the > window expiry. By persisting the window transparently in the state backend > and caching/iterating them as need, we could support larger windows and also > windowed bolts with user/application state. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
