[jira] [Created] (STORM-2084) after supervisor v2 merge async localizer and localizer
Robert Joseph Evans created STORM-2084: -- Summary: after supervisor v2 merge async localizer and localizer Key: STORM-2084 URL: https://issues.apache.org/jira/browse/STORM-2084 Project: Apache Storm Issue Type: Improvement Components: storm-core Affects Versions: 2.0.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Once we mere in STORM-2018 https://github.com/apache/storm/pull/1642 we should look into merging the two localizers into a single class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-2054) DependencyResolver should be aware of relative path and absolute path
[ https://issues.apache.org/jira/browse/STORM-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465719#comment-15465719 ] Robert Joseph Evans commented on STORM-2054: I merged the pull request to master, but it looks like there are merge conflicts on 1.x and I assume 1.1.x too. > DependencyResolver should be aware of relative path and absolute path > - > > Key: STORM-2054 > URL: https://issues.apache.org/jira/browse/STORM-2054 > Project: Apache Storm > Issue Type: Bug > Components: storm-submit-tools >Affects Versions: 1.1.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Critical > Time Spent: 1h 20m > Remaining Estimate: 0h > > DependencyResolver always create directory based on storm.home or current > working directory which is intended for relative path but not intended for > absolute path. > Furthermore, DependencyResolverTest doesn't remove temporary directory after > testing. Test creates a new temporary absolute path but due to this bug, > temporary directory is created in working directory which prevents cleaning > up, and finally making RAT error on all builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-2071) nimbus-test test-leadership failing with Exception
[
https://issues.apache.org/jira/browse/STORM-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452246#comment-15452246
]
Robert Joseph Evans commented on STORM-2071:
I replaced the sleep with
{code}
(defn wait-for-status [nimbus name status]
(while-timeout 5000
(let [topo-summary (first (filter (fn [topo] (= name (.get_name topo)))
(.get_topologies (.getClusterInfo nimbus
topo-status (if topo-summary (.get_status topo-summary)
"NOT-RUNNING")]
(log-message "WAITING FOR "name" TO BE " status " CURRENT " topo-status)
(not= topo-status status))
(Thread/sleep 100)))
...
(wait-for-status nimbus "t1" "ACTIVE")
{code}
If someone wants to put this in a separate pull request or have me break it off
I am happy to do it.
> nimbus-test test-leadership failing with Exception
> --
>
> Key: STORM-2071
> URL: https://issues.apache.org/jira/browse/STORM-2071
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0, 1.0.1
> Environment: Mac os X
>Reporter: Paul Poulosky
>Assignee: Robert Joseph Evans
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When running unit tests on my Mac, I get repeated failures in test-leadership.
>
> 73752 [main] INFO o.a.s.l.ThriftAccessLogger - Request ID: 1 access from:
> null principal: null operation: deactivate
> ]]>
> Uncaught
> exception, not in assertion.
> expected: nil
> actual: java.lang.RuntimeException: No transition for event: :inactivate,
> status: {:type :rebalancing} storm-id: t1-1-1472598899
> at org.apache.storm.daemon.nimbus$transition_BANG_$get_event__4879.invoke
> (nimbus.clj:365)
> org.apache.storm.daemon.nimbus$transition_BANG_.invoke (nimbus.clj:373)
> clojure.lang.AFn.applyToHelper (AFn.java:165)
> clojure.lang.AFn.applyTo (AFn.java:144)
> clojure.core$apply.invoke (core.clj:636)
> org.apache.storm.daemon.nimbus$transition_name_BANG_.doInvoke
> (nimbus.clj:391)
> clojure.lang.RestFn.invoke (RestFn.java:467)
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__5850.deactivate
> (nimbus.clj:1773)
> sun.reflect.NativeMethodAccessorImpl.invoke0
> (NativeMethodAccessorImpl.java:-2)
> sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke (Method.java:497)
> clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
> clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
> org.apache.storm.nimbus_test$fn__1203$fn__1209.invoke
> (nimbus_test.clj:1222)
> org.apache.storm.nimbus_test/fn (nimbus_test.clj:1210)
> clojure.test$test_var$fn__7670.invoke (test.clj:704)
> clojure.test$test_var.invoke (test.clj:704)
> clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars$fn__7692.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars.invoke (test.clj:718)
> clojure.test$test_all_vars.invoke (test.clj:728)
> (test.clj:747)
> clojure.core$map$fn__4553.invoke (core.clj:2624)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.Cons.next (Cons.java:39)
> clojure.lang.RT.boundedLength (RT.java:1735)
> clojure.lang.RestFn.applyTo (RestFn.java:130)
> clojure.core$apply.invoke (core.clj:632)
> clojure.test$run_tests.doInvoke (test.clj:762)
> clojure.lang.RestFn.invoke (RestFn.java:408)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365$fn__8366.invoke
> (test_runner.clj:107)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365.invoke
> (test_runner.clj:53)
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364.invoke
> (test_runner.clj:52)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.RT.seq (RT.java:507)
> clojure.core/seq (core.clj:137)
> clojure.core$dorun.invoke (core.clj:3009)
> org.apache.storm.testrunner$eval8358.invoke (test_runner.clj:52)
> clojure.lang.Compiler.eval (Compiler.java:6782)
> clojure.lang.Compiler.load (Compiler.java:7227)
> clojure.lang.Compiler.loadFile (Compiler.java:7165)
> clojure.main$load_script.invoke (main.clj:275)
> clojure.main$script_opt.invoke (main.clj:337)
> clojure.main$main.doInvoke (main.clj:421)
> clojure.lang.RestFn.invoke (RestFn.java:421)
> clojure.lang.Var.invoke (Var.java:383)
> clojure.lang.AFn.applyToHelper (AFn.java:156)
>
[jira] [Commented] (STORM-2071) nimbus-test test-leadership failing with Exception
[
https://issues.apache.org/jira/browse/STORM-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452239#comment-15452239
]
Robert Joseph Evans commented on STORM-2071:
I found and fixed this in my supervisor v2 patch. I fixed a few bugs because I
was not sure if they were caused by the supervisor changes or not.
https://github.com/apache/storm/pull/1642
> nimbus-test test-leadership failing with Exception
> --
>
> Key: STORM-2071
> URL: https://issues.apache.org/jira/browse/STORM-2071
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0, 1.0.1
> Environment: Mac os X
>Reporter: Paul Poulosky
>Assignee: Robert Joseph Evans
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When running unit tests on my Mac, I get repeated failures in test-leadership.
>
> 73752 [main] INFO o.a.s.l.ThriftAccessLogger - Request ID: 1 access from:
> null principal: null operation: deactivate
> ]]>
> Uncaught
> exception, not in assertion.
> expected: nil
> actual: java.lang.RuntimeException: No transition for event: :inactivate,
> status: {:type :rebalancing} storm-id: t1-1-1472598899
> at org.apache.storm.daemon.nimbus$transition_BANG_$get_event__4879.invoke
> (nimbus.clj:365)
> org.apache.storm.daemon.nimbus$transition_BANG_.invoke (nimbus.clj:373)
> clojure.lang.AFn.applyToHelper (AFn.java:165)
> clojure.lang.AFn.applyTo (AFn.java:144)
> clojure.core$apply.invoke (core.clj:636)
> org.apache.storm.daemon.nimbus$transition_name_BANG_.doInvoke
> (nimbus.clj:391)
> clojure.lang.RestFn.invoke (RestFn.java:467)
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__5850.deactivate
> (nimbus.clj:1773)
> sun.reflect.NativeMethodAccessorImpl.invoke0
> (NativeMethodAccessorImpl.java:-2)
> sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke (Method.java:497)
> clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
> clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
> org.apache.storm.nimbus_test$fn__1203$fn__1209.invoke
> (nimbus_test.clj:1222)
> org.apache.storm.nimbus_test/fn (nimbus_test.clj:1210)
> clojure.test$test_var$fn__7670.invoke (test.clj:704)
> clojure.test$test_var.invoke (test.clj:704)
> clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars$fn__7692.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars.invoke (test.clj:718)
> clojure.test$test_all_vars.invoke (test.clj:728)
> (test.clj:747)
> clojure.core$map$fn__4553.invoke (core.clj:2624)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.Cons.next (Cons.java:39)
> clojure.lang.RT.boundedLength (RT.java:1735)
> clojure.lang.RestFn.applyTo (RestFn.java:130)
> clojure.core$apply.invoke (core.clj:632)
> clojure.test$run_tests.doInvoke (test.clj:762)
> clojure.lang.RestFn.invoke (RestFn.java:408)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365$fn__8366.invoke
> (test_runner.clj:107)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365.invoke
> (test_runner.clj:53)
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364.invoke
> (test_runner.clj:52)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.RT.seq (RT.java:507)
> clojure.core/seq (core.clj:137)
> clojure.core$dorun.invoke (core.clj:3009)
> org.apache.storm.testrunner$eval8358.invoke (test_runner.clj:52)
> clojure.lang.Compiler.eval (Compiler.java:6782)
> clojure.lang.Compiler.load (Compiler.java:7227)
> clojure.lang.Compiler.loadFile (Compiler.java:7165)
> clojure.main$load_script.invoke (main.clj:275)
> clojure.main$script_opt.invoke (main.clj:337)
> clojure.main$main.doInvoke (main.clj:421)
> clojure.lang.RestFn.invoke (RestFn.java:421)
> clojure.lang.Var.invoke (Var.java:383)
> clojure.lang.AFn.applyToHelper (AFn.java:156)
> clojure.lang.Var.applyTo (Var.java:700)
> clojure.main.main (main.java:37)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (STORM-2071) nimbus-test test-leadership failing with Exception
[
https://issues.apache.org/jira/browse/STORM-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reassigned STORM-2071:
--
Assignee: Robert Joseph Evans
> nimbus-test test-leadership failing with Exception
> --
>
> Key: STORM-2071
> URL: https://issues.apache.org/jira/browse/STORM-2071
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0, 1.0.1
> Environment: Mac os X
>Reporter: Paul Poulosky
>Assignee: Robert Joseph Evans
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When running unit tests on my Mac, I get repeated failures in test-leadership.
>
> 73752 [main] INFO o.a.s.l.ThriftAccessLogger - Request ID: 1 access from:
> null principal: null operation: deactivate
> ]]>
> Uncaught
> exception, not in assertion.
> expected: nil
> actual: java.lang.RuntimeException: No transition for event: :inactivate,
> status: {:type :rebalancing} storm-id: t1-1-1472598899
> at org.apache.storm.daemon.nimbus$transition_BANG_$get_event__4879.invoke
> (nimbus.clj:365)
> org.apache.storm.daemon.nimbus$transition_BANG_.invoke (nimbus.clj:373)
> clojure.lang.AFn.applyToHelper (AFn.java:165)
> clojure.lang.AFn.applyTo (AFn.java:144)
> clojure.core$apply.invoke (core.clj:636)
> org.apache.storm.daemon.nimbus$transition_name_BANG_.doInvoke
> (nimbus.clj:391)
> clojure.lang.RestFn.invoke (RestFn.java:467)
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__5850.deactivate
> (nimbus.clj:1773)
> sun.reflect.NativeMethodAccessorImpl.invoke0
> (NativeMethodAccessorImpl.java:-2)
> sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke (Method.java:497)
> clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
> clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
> org.apache.storm.nimbus_test$fn__1203$fn__1209.invoke
> (nimbus_test.clj:1222)
> org.apache.storm.nimbus_test/fn (nimbus_test.clj:1210)
> clojure.test$test_var$fn__7670.invoke (test.clj:704)
> clojure.test$test_var.invoke (test.clj:704)
> clojure.test$test_vars$fn__7692$fn__7697.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars$fn__7692.invoke (test.clj:722)
> clojure.test$default_fixture.invoke (test.clj:674)
> clojure.test$test_vars.invoke (test.clj:718)
> clojure.test$test_all_vars.invoke (test.clj:728)
> (test.clj:747)
> clojure.core$map$fn__4553.invoke (core.clj:2624)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.Cons.next (Cons.java:39)
> clojure.lang.RT.boundedLength (RT.java:1735)
> clojure.lang.RestFn.applyTo (RestFn.java:130)
> clojure.core$apply.invoke (core.clj:632)
> clojure.test$run_tests.doInvoke (test.clj:762)
> clojure.lang.RestFn.invoke (RestFn.java:408)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365$fn__8366.invoke
> (test_runner.clj:107)
>
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364$fn__8365.invoke
> (test_runner.clj:53)
> org.apache.storm.testrunner$eval8358$iter__8359__8363$fn__8364.invoke
> (test_runner.clj:52)
> clojure.lang.LazySeq.sval (LazySeq.java:40)
> clojure.lang.LazySeq.seq (LazySeq.java:49)
> clojure.lang.RT.seq (RT.java:507)
> clojure.core/seq (core.clj:137)
> clojure.core$dorun.invoke (core.clj:3009)
> org.apache.storm.testrunner$eval8358.invoke (test_runner.clj:52)
> clojure.lang.Compiler.eval (Compiler.java:6782)
> clojure.lang.Compiler.load (Compiler.java:7227)
> clojure.lang.Compiler.loadFile (Compiler.java:7165)
> clojure.main$load_script.invoke (main.clj:275)
> clojure.main$script_opt.invoke (main.clj:337)
> clojure.main$main.doInvoke (main.clj:421)
> clojure.lang.RestFn.invoke (RestFn.java:421)
> clojure.lang.Var.invoke (Var.java:383)
> clojure.lang.AFn.applyToHelper (AFn.java:156)
> clojure.lang.Var.applyTo (Var.java:700)
> clojure.main.main (main.java:37)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1344) storm-jdbc build error "object name already exists: USER_DETAILS in statement"
[ https://issues.apache.org/jira/browse/STORM-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452218#comment-15452218 ] Robert Joseph Evans commented on STORM-1344: Do you have a patch for that? I can see it being at least a step in the right direction on fixing this. > storm-jdbc build error "object name already exists: USER_DETAILS in statement" > -- > > Key: STORM-1344 > URL: https://issues.apache.org/jira/browse/STORM-1344 > Project: Apache Storm > Issue Type: Bug > Components: storm-jdbc >Affects Versions: 1.0.0 > Environment: os X, jdk7 >Reporter: Longda Feng >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > ``` > [ERROR] Failed to execute goal org.codehaus.mojo:sql-maven-plugin:1.5:execute > (create-db) on project storm-jdbc: object name already exists: USER_DETAILS > in statement [ /** * Licensed to the Apache Software Foundation (ASF) under > one * or more contributor license agreements. See the NOTICE file * > distributed with this work for additional information * regarding copyright > ownership. The ASF licenses this file * to you under the Apache License, > Version 2.0 (the * "License"); you may not use this file except in compliance > * with the License. You may obtain a copy of the License at * * > http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable > law or agreed to in writing, software * distributed under the License is > distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY > KIND, either express or implied. * See the License for the specific language > governing permissions and * limitations under the License. */ > [ERROR] create table user_details (id integer, user_name varchar(100), > create_date date)] > [ERROR] -> [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.codehaus.mojo:sql-maven-plugin:1.5:execute (create-db) on project > storm-jdbc: object name already exists: USER_DETAILS in statement [ /** * > Licensed to the Apache Software Foundation (ASF) under one * or more > contributor license agreements. See the NOTICE file * distributed with this > work for additional information * regarding copyright ownership. The ASF > licenses this file * to you under the Apache License, Version 2.0 (the * > "License"); you may not use this file except in compliance * with the > License. You may obtain a copy of the License at * * > http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable > law or agreed to in writing, software * distributed under the License is > distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY > KIND, either express or implied. * See the License for the specific language > governing permissions and * limitations under the License. */ > create table user_details (id integer, user_name varchar(100), create_date > date)] > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:197) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > at >
[jira] [Commented] (STORM-1985) Provide a tool for showing and killing corrupted topology
[ https://issues.apache.org/jira/browse/STORM-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449339#comment-15449339 ] Robert Joseph Evans commented on STORM-1985: Please put the class in org.apache.storm.command. A pull request against https://github.com/apache/storm is preferable to posting a patch in the JIRA. Just make sure to include the jira number (STORM-1985) in the title of the pull request. There are a number of design/usability choices that I would like to see fixed. https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/command/Blobstore.java Has a better framework for having a single command with multiple sub commands. I would prefer to see admin be similar, having a special hard coded string with spaces in it does not fit with the rest of storm. The code you posed is not complete and has a big TODO in it. The code that you do have looks correct. > Provide a tool for showing and killing corrupted topology > - > > Key: STORM-1985 > URL: https://issues.apache.org/jira/browse/STORM-1985 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Jungtaek Lim >Assignee: Kamal > Labels: newbie > Attachments: AdminCommands.java, proposal_admin_tool_design.docx > > > After STORM-1976, Nimbus doesn't clean up corrupted topologies. > (corrupted topology means the topology whose codes are not available on > blobstore.) > Also after STORM-1977, no Nimbus is gaining leadership if one or more > topologies are corrupted, which means all nimbuses will be no-op. > So we should provide a tool to kill specific topology without accessing > leader nimbus (because there's no leader nimbus at that time). The tool > should also determine which topologies are corrupted, and show its list or > clean up automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-2059) storm-submit-tools is getting rat failures.
[
https://issues.apache.org/jira/browse/STORM-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439171#comment-15439171
]
Robert Joseph Evans commented on STORM-2059:
[~kabhwan],
Could you take a look at this. STORM-2016 caused this. Although the pull
requests for it are a bit confusing to me.
Also what I see from running RAT is something really odd, because somehow it is
scanning a tmp directory too (on mac it is a really odd tmp directory before
getting to the dr-test*).
{code}
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/commons-cli/commons-cli/1.2/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/commons-cli/commons-cli/1.2/commons-cli-1.2.jar.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/commons-cli/commons-cli/1.2/commons-cli-1.2.pom.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/apache/10/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/apache/10/apache-10.pom.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/apache/4/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/apache/4/apache-4.pom.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/commons/commons-parent/11/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/commons/commons-parent/11/commons-parent-11.pom.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/flux/1.0.0/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/flux/1.0.0/flux-1.0.0.pom.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/flux-core/1.0.0/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/flux-core/1.0.0/flux-core-1.0.0.jar.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/flux-core/1.0.0/flux-core-1.0.0.pom
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/flux-core/1.0.0/flux-core-1.0.0.pom.sha1
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/storm/1.0.0/_maven.repositories
var/folders/tw/30gm5zz130xb7065nq53wh0m0015wb/T/dr-test2099561184234297394/org/apache/storm/storm/1.0.0/storm-1.0.0.pom.sha1
{code}
> storm-submit-tools is getting rat failures.
> ---
>
> Key: STORM-2059
> URL: https://issues.apache.org/jira/browse/STORM-2059
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-submit-tools
>Reporter: Robert Joseph Evans
> Fix For: 2.0.0
>
>
> https://travis-ci.org/revans2/incubator-storm/jobs/155187695
> {code}
> [INFO] Rat check: Summary of files. Unapproved: 17 unknown: 17 generated: 0
> approved: 14 licence.
> ...
> [INFO] storm-submit-tools . FAILURE [ 2.296
> s]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-2059) storm-submit-tools is getting rat failures.
Robert Joseph Evans created STORM-2059:
--
Summary: storm-submit-tools is getting rat failures.
Key: STORM-2059
URL: https://issues.apache.org/jira/browse/STORM-2059
Project: Apache Storm
Issue Type: Bug
Components: storm-submit-tools
Reporter: Robert Joseph Evans
Fix For: 2.0.0
https://travis-ci.org/revans2/incubator-storm/jobs/155187695
{code}
[INFO] Rat check: Summary of files. Unapproved: 17 unknown: 17 generated: 0
approved: 14 licence.
...
[INFO] storm-submit-tools . FAILURE [ 2.296 s]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (STORM-1903) Intermittent Travis test failures in org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
[
https://issues.apache.org/jira/browse/STORM-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reassigned STORM-1903:
--
Assignee: Robert Joseph Evans
> Intermittent Travis test failures in
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
>
>
> Key: STORM-1903
> URL: https://issues.apache.org/jira/browse/STORM-1903
> Project: Apache Storm
> Issue Type: Test
> Components: storm-core
>Reporter: Abhishek Agarwal
>Assignee: Robert Joseph Evans
>
> testSubmitTopologyToLocalNimbus fails with following error
> {noformat}
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
> at
> org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
> at org.apache.storm.utils.Utils.getTopologyInfo(Utils.java:1465)
> at org.apache.storm.LocalCluster$submit_hook.invoke(LocalCluster.clj:44)
> at
> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:52)
> at org.apache.storm.LocalCluster.submitTopology(Unknown Source)
> at
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus(LocalNimbusTest.java:65)
> {noformat}
> The exception on server is
> {noformat}
> 283607 [pool-75-thread-3] ERROR o.a.t.s.AbstractNonblockingServer$FrameBuffer
> - Unexpected throwable while invoking!
> java.lang.NullPointerException
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26)
> ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758$iter__4869__4873$fn__4874.invoke(nimbus.clj:1870)
> ~[classes/:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
> at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfoWithOpts(nimbus.clj:1868)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfo(nimbus.clj:1907)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3748)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3732)
> ~[classes/:?]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> ~[classes/:?]
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> [libthrift-0.9.3.jar:0.9.3]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1903) Intermittent Travis test failures in org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
[
https://issues.apache.org/jira/browse/STORM-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433393#comment-15433393
]
Robert Joseph Evans commented on STORM-1903:
Looks like there are cases where beats which comes from the heart beat cache
can be nil. We are checking other places for the values in it if they are nil,
but not beats itself. It should be a simple one line fix.
> Intermittent Travis test failures in
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
>
>
> Key: STORM-1903
> URL: https://issues.apache.org/jira/browse/STORM-1903
> Project: Apache Storm
> Issue Type: Test
> Components: storm-core
>Reporter: Abhishek Agarwal
>
> testSubmitTopologyToLocalNimbus fails with following error
> {noformat}
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
> at
> org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
> at org.apache.storm.utils.Utils.getTopologyInfo(Utils.java:1465)
> at org.apache.storm.LocalCluster$submit_hook.invoke(LocalCluster.clj:44)
> at
> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:52)
> at org.apache.storm.LocalCluster.submitTopology(Unknown Source)
> at
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus(LocalNimbusTest.java:65)
> {noformat}
> The exception on server is
> {noformat}
> 283607 [pool-75-thread-3] ERROR o.a.t.s.AbstractNonblockingServer$FrameBuffer
> - Unexpected throwable while invoking!
> java.lang.NullPointerException
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26)
> ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758$iter__4869__4873$fn__4874.invoke(nimbus.clj:1870)
> ~[classes/:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
> at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfoWithOpts(nimbus.clj:1868)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfo(nimbus.clj:1907)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3748)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3732)
> ~[classes/:?]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> ~[classes/:?]
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> [libthrift-0.9.3.jar:0.9.3]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1903) Intermittent Travis test failures in org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
[
https://issues.apache.org/jira/browse/STORM-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433073#comment-15433073
]
Robert Joseph Evans commented on STORM-1903:
Actually I am wrong, the NPE indicates that the connection was up, but for some
reason nimbus had an issue with getting the cluster info and there was
something null in the data structure.
> Intermittent Travis test failures in
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
>
>
> Key: STORM-1903
> URL: https://issues.apache.org/jira/browse/STORM-1903
> Project: Apache Storm
> Issue Type: Test
> Components: storm-core
>Reporter: Abhishek Agarwal
>
> testSubmitTopologyToLocalNimbus fails with following error
> {noformat}
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
> at
> org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
> at org.apache.storm.utils.Utils.getTopologyInfo(Utils.java:1465)
> at org.apache.storm.LocalCluster$submit_hook.invoke(LocalCluster.clj:44)
> at
> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:52)
> at org.apache.storm.LocalCluster.submitTopology(Unknown Source)
> at
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus(LocalNimbusTest.java:65)
> {noformat}
> The exception on server is
> {noformat}
> 283607 [pool-75-thread-3] ERROR o.a.t.s.AbstractNonblockingServer$FrameBuffer
> - Unexpected throwable while invoking!
> java.lang.NullPointerException
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26)
> ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758$iter__4869__4873$fn__4874.invoke(nimbus.clj:1870)
> ~[classes/:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
> at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfoWithOpts(nimbus.clj:1868)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfo(nimbus.clj:1907)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3748)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3732)
> ~[classes/:?]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> ~[classes/:?]
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> [libthrift-0.9.3.jar:0.9.3]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1903) Intermittent Travis test failures in org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
[
https://issues.apache.org/jira/browse/STORM-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433007#comment-15433007
]
Robert Joseph Evans commented on STORM-1903:
Please don't put in a sleep, we need a way to tell if it is bound or not,
because the test already takes over 30 seconds, adding in more delays is far
from ideal.
> Intermittent Travis test failures in
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
>
>
> Key: STORM-1903
> URL: https://issues.apache.org/jira/browse/STORM-1903
> Project: Apache Storm
> Issue Type: Test
> Components: storm-core
>Reporter: Abhishek Agarwal
>
> testSubmitTopologyToLocalNimbus fails with following error
> {noformat}
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
> at
> org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
> at org.apache.storm.utils.Utils.getTopologyInfo(Utils.java:1465)
> at org.apache.storm.LocalCluster$submit_hook.invoke(LocalCluster.clj:44)
> at
> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:52)
> at org.apache.storm.LocalCluster.submitTopology(Unknown Source)
> at
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus(LocalNimbusTest.java:65)
> {noformat}
> The exception on server is
> {noformat}
> 283607 [pool-75-thread-3] ERROR o.a.t.s.AbstractNonblockingServer$FrameBuffer
> - Unexpected throwable while invoking!
> java.lang.NullPointerException
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26)
> ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758$iter__4869__4873$fn__4874.invoke(nimbus.clj:1870)
> ~[classes/:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
> at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfoWithOpts(nimbus.clj:1868)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfo(nimbus.clj:1907)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3748)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3732)
> ~[classes/:?]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> ~[classes/:?]
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> [libthrift-0.9.3.jar:0.9.3]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1903) Intermittent Travis test failures in org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
[
https://issues.apache.org/jira/browse/STORM-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433002#comment-15433002
]
Robert Joseph Evans commented on STORM-1903:
I think it much be a race between when the nimbus daemon is up and ready to
serve requests, and at what point we submit the topology.
> Intermittent Travis test failures in
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus
>
>
> Key: STORM-1903
> URL: https://issues.apache.org/jira/browse/STORM-1903
> Project: Apache Storm
> Issue Type: Test
> Components: storm-core
>Reporter: Abhishek Agarwal
>
> testSubmitTopologyToLocalNimbus fails with following error
> {noformat}
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> at
> org.apache.storm.generated.Nimbus$Client.recv_getTopologyInfo(Nimbus.java:1182)
> at
> org.apache.storm.generated.Nimbus$Client.getTopologyInfo(Nimbus.java:1169)
> at org.apache.storm.utils.Utils.getTopologyInfo(Utils.java:1465)
> at org.apache.storm.LocalCluster$submit_hook.invoke(LocalCluster.clj:44)
> at
> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:52)
> at org.apache.storm.LocalCluster.submitTopology(Unknown Source)
> at
> org.apache.storm.nimbus.LocalNimbusTest.testSubmitTopologyToLocalNimbus(LocalNimbusTest.java:65)
> {noformat}
> The exception on server is
> {noformat}
> 283607 [pool-75-thread-3] ERROR o.a.t.s.AbstractNonblockingServer$FrameBuffer
> - Unexpected throwable while invoking!
> java.lang.NullPointerException
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26)
> ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758$iter__4869__4873$fn__4874.invoke(nimbus.clj:1870)
> ~[classes/:?]
> at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
> at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
> at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
> at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
> at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
> at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfoWithOpts(nimbus.clj:1868)
> ~[classes/:?]
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__4758.getTopologyInfo(nimbus.clj:1907)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3748)
> ~[classes/:?]
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfo.getResult(Nimbus.java:3732)
> ~[classes/:?]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> ~[classes/:?]
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> [libthrift-0.9.3.jar:0.9.3]
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> [libthrift-0.9.3.jar:0.9.3]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_31]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1985) Provide a tool for showing and killing corrupted topology
[ https://issues.apache.org/jira/browse/STORM-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430751#comment-15430751 ] Robert Joseph Evans commented on STORM-1985: I personally would prefer a tool that pretends to be Nimbus and acts like Nimbus. Meaning it would connect to zookeeper, blobstore, local caches, etc. just as if it were the nimbus daemon. We have run into situations in the past where nimbus is down because of bad state stored somewhere. Having a tool that can do everything nimbus does is important. Having a separate daemon to do this feels too complicated, and also exposes a lot more potential for attack. At the beginning I would say just have a command line tool that will create a [ClusterState|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/cluster/IStormClusterState.java] a [BlobStore|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/blobstore/BlobStore.java] and possibly a [LocalState|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/utils/LocalState.java] like nimbus currently does. Once those are created for this project we would just then run through some code very similar to [cleanup-corrupt-topologies!|https://github.com/apache/storm/pull/1572/files]. In the future we could have it do many more things. Having a UI in the future would probably need a separate daemon for security reasons, but we use this type of operation so rarely that I don't see much value in setting up an RPC daemon for it. If we want a UI have the UI be baked into the admin command so it would be a web process that is running with the same privlages as nimbus, and there is no need for RPC at all, just run it locally. > Provide a tool for showing and killing corrupted topology > - > > Key: STORM-1985 > URL: https://issues.apache.org/jira/browse/STORM-1985 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Jungtaek Lim >Assignee: Kamal > Labels: newbie > Attachments: proposal_admin_tool_design.docx > > > After STORM-1976, Nimbus doesn't clean up corrupted topologies. > (corrupted topology means the topology whose codes are not available on > blobstore.) > Also after STORM-1977, no Nimbus is gaining leadership if one or more > topologies are corrupted, which means all nimbuses will be no-op. > So we should provide a tool to kill specific topology without accessing > leader nimbus (because there's no leader nimbus at that time). The tool > should also determine which topologies are corrupted, and show its list or > clean up automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1985) Provide a tool for showing and killing corrupted topology
[ https://issues.apache.org/jira/browse/STORM-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-1985: --- Assignee: Kamal > Provide a tool for showing and killing corrupted topology > - > > Key: STORM-1985 > URL: https://issues.apache.org/jira/browse/STORM-1985 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Jungtaek Lim >Assignee: Kamal > Labels: newbie > Attachments: proposal_admin_tool_design.docx > > > After STORM-1976, Nimbus doesn't clean up corrupted topologies. > (corrupted topology means the topology whose codes are not available on > blobstore.) > Also after STORM-1977, no Nimbus is gaining leadership if one or more > topologies are corrupted, which means all nimbuses will be no-op. > So we should provide a tool to kill specific topology without accessing > leader nimbus (because there's no leader nimbus at that time). The tool > should also determine which topologies are corrupted, and show its list or > clean up automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1434) Support the GROUP BY clause in StormSQL
[ https://issues.apache.org/jira/browse/STORM-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422709#comment-15422709 ] Robert Joseph Evans commented on STORM-1434: Streaming joins/group-by really only make since in the context of a window (large or small). I personally would look towards BEAM and how they handle this, If you try to do a GROUP BY of any kind outside of a window it results in an error. If you are doing a rolling or tumbling window it gets a bit more complex, and because of that complexity I think we should look towards others that have done something like this already. As for the aggregation after the group-by that really depends on the aggregation. Ones where we can checkpoint a partial aggregation instead of all of the source values would be ideal, but it is an optimization over a more general aggregation that we probably also want to support. > Support the GROUP BY clause in StormSQL > --- > > Key: STORM-1434 > URL: https://issues.apache.org/jira/browse/STORM-1434 > Project: Apache Storm > Issue Type: New Feature > Components: storm-sql >Reporter: Haohui Mai > > This jira tracks the effort of implement the support `GROUP BY` clause in > StormSQL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1985) Provide a tool for showing and killing corrupted topology
[ https://issues.apache.org/jira/browse/STORM-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421384#comment-15421384 ] Robert Joseph Evans commented on STORM-1985: I would start off with command line, if you are feeling ambitious you could look at adding a UI component to it. > Provide a tool for showing and killing corrupted topology > - > > Key: STORM-1985 > URL: https://issues.apache.org/jira/browse/STORM-1985 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Jungtaek Lim > Labels: newbie > > After STORM-1976, Nimbus doesn't clean up corrupted topologies. > (corrupted topology means the topology whose codes are not available on > blobstore.) > Also after STORM-1977, no Nimbus is gaining leadership if one or more > topologies are corrupted, which means all nimbuses will be no-op. > So we should provide a tool to kill specific topology without accessing > leader nimbus (because there's no leader nimbus at that time). The tool > should also determine which topologies are corrupted, and show its list or > clean up automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (STORM-2038) Provide an alternative to using symlinks
[
https://issues.apache.org/jira/browse/STORM-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421036#comment-15421036
]
Robert Joseph Evans edited comment on STORM-2038 at 8/15/16 2:23 PM:
-
Giving a canonical path to the worker artifacts should be a fairly simple
solution. We were doing it previously for the logs dir anyways, it should be
simple to extend this and just disable the symlink when configured to do so.
For the blob store we have a bit of a bigger problem. The
[Localizer|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/localizer/Localizer.java]
sets up a chain of symlinks so that the old data downloaded from the blob
store can remain in place until the new data is downloaded and ready. At that
point it will update one of the sym-links in the chain to atomically point it
to the new location of the data. There is some redundancy in the links that we
could probably remove, but the path currently stands as
{code}
${worker_pwd}/${link_name} -> ${topology_code_dir}/${link_name} ->
${localizer_cache}/${user}/.../${key}.current ->
${localizer_cache}/${user}/.../${key}.${version}
{code}
If we removed all of the symlinks in some cases we would need another way/API
for the user to be able to get the current list of blob paths to access. We
currently don't have a communication path from the supervisor to the worker.
We would need to add this in, along with some book keeping so we can know which
blob version is the current one. We don't always rely on the version number to
be atomically incrementing, just different from what we already have cached.
Any high level API that we do add, would need to work both with sym-links and
without sym-links consistently. Essentially it would need two implementations
one that relies on sym-links so when a sym-link changes the API returns the
correct thing, and another that just reads from this new communication path.
There are a number of other features in the works that build on top of this
functionality that would also need some rework. STORM-2016 takes jars on the
client and adds them to the blobstore/classpath for the worker (removes the
requirement for an uber-jar).
I also know that [~jerrypeng] has been working on a few things that would allow
you to change configs as part of a topology rebalance, although it is very
preliminary. It also has the potential to also update a topology's jar, or
combined with STORM-2016 a dependency of a topology and upgrade the topology on
the fly without actually relaunching it.
None of this makes this work impossible, just not trivial.
was (Author: revans2):
Giving a canonical path to the worker artifacts should be a fairly simple
solution. We were doing it previously for the logs dir anyways, it should be
simple to extend this and just disable the symlink when configured to do so.
For the blob store we have a bit of a bigger problem. The
[Localizer|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/localizer/Localizer.java]
sets up a chain of symlinks so that the old data downloaded from the blob
store can remain in place until the new data is downloaded and ready. At that
point it will update one of the sym-links in the chain to atomically point it
to the new location of the data. There is some redundancy in the links that we
could probably remove, but the path currently stands as
{code}
${worker_pwd}/link_name -> ${topology_code_dir}/link_name ->
${localizer_cache}/${user}/.../${key}.current ->
${localizer_cache}/${user}/.../${key}.${version}
{code}
If we removed all of the symlinks in some cases we would need another way/API
for the user to be able to get the current list of blob paths to access. We
currently don't have a communication path from the supervisor to the worker.
We would need to add this in, along with some book keeping so we can know which
blob version is the current one. We don't always rely on the version number to
be atomically incrementing, just different from what we already have cached.
Any high level API that we do add, would need to work both with sym-links and
without sym-links consistently. Essentially it would need two implementations
one that relies on sym-links so when a sym-link changes the API returns the
correct thing, and another that just reads from this new communication path.
There are a number of other features in the works that build on top of this
functionality that would also need some rework. STORM-2016 takes jars on the
client and adds them to the blobstore/classpath for the worker (removes the
requirement for an uber-jar).
I also know that [~jerrypeng] has been working on a few things that would allow
you to change configs as part of a topology rebalance, although it is very
preliminary. It also has the potential to also update a topology's
[jira] [Commented] (STORM-2038) Provide an alternative to using symlinks
[
https://issues.apache.org/jira/browse/STORM-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421036#comment-15421036
]
Robert Joseph Evans commented on STORM-2038:
Giving a canonical path to the worker artifacts should be a fairly simple
solution. We were doing it previously for the logs dir anyways, it should be
simple to extend this and just disable the symlink when configured to do so.
For the blob store we have a bit of a bigger problem. The
[Localizer|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/localizer/Localizer.java]
sets up a chain of symlinks so that the old data downloaded from the blob
store can remain in place until the new data is downloaded and ready. At that
point it will update one of the sym-links in the chain to atomically point it
to the new location of the data. There is some redundancy in the links that we
could probably remove, but the path currently stands as
{code}
${worker_pwd}/link_name -> ${topology_code_dir}/link_name ->
${localizer_cache}/${user}/.../${key}.current ->
${localizer_cache}/${user}/.../${key}.${version}
{code}
If we removed all of the symlinks in some cases we would need another way/API
for the user to be able to get the current list of blob paths to access. We
currently don't have a communication path from the supervisor to the worker.
We would need to add this in, along with some book keeping so we can know which
blob version is the current one. We don't always rely on the version number to
be atomically incrementing, just different from what we already have cached.
Any high level API that we do add, would need to work both with sym-links and
without sym-links consistently. Essentially it would need two implementations
one that relies on sym-links so when a sym-link changes the API returns the
correct thing, and another that just reads from this new communication path.
There are a number of other features in the works that build on top of this
functionality that would also need some rework. STORM-2016 takes jars on the
client and adds them to the blobstore/classpath for the worker (removes the
requirement for an uber-jar).
I also know that [~jerrypeng] has been working on a few things that would allow
you to change configs as part of a topology rebalance, although it is very
preliminary. It also has the potential to also update a topology's jar, or
combined with STORM-2016 a dependency of a topology and upgrade the topology on
the fly without actually relaunching it.
None of this makes this work impossible, just not trivial.
> Provide an alternative to using symlinks
>
>
> Key: STORM-2038
> URL: https://issues.apache.org/jira/browse/STORM-2038
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
>Affects Versions: 1.0.1
> Environment: Any windows
>Reporter: Paul Milliken
> Labels: symlink, windows
>
> As of Storm 1.0 and above, some functionality (such as the worker-artifacts
> directory) require the use of symlinks. On Windows platforms, this requires
> that Storm either be run as an administrator or that certain group policy
> settings are changed.
> In locked-down environments, both of these solutions are not suitable.
> Where possible, an alternative option should be provided to the use of
> symlinks. For example, it may be possible to create additional copies of the
> worker artifacts directory for each worker (possibly inefficient) or provide
> the workers with the canonical path to the real directory.
> See the [brief
> discussion|http://mail-archives.apache.org/mod_mbox/storm-dev/201608.mbox/%3C1293850887.13165119.1471022901569.JavaMail.yahoo%40mail.yahoo.com%3E]
> on the mailing list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1985) Provide a tool for showing and killing corrupted topology
[ https://issues.apache.org/jira/browse/STORM-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420963#comment-15420963 ] Robert Joseph Evans commented on STORM-1985: It was asked on the mailing list where this code should live. This was my answer. bq. My guess is that this is going to be a new tool. I think we want to start looking into supporting a storm admin tool that will give users that have access to super user credentials access to directly examine/modify the state of a running storm cluster, zookeeper, blobstore, and local state. I don't expect you to do all of that work, but at least starting a place for a storm admin command in my opinion is the best place for this. > Provide a tool for showing and killing corrupted topology > - > > Key: STORM-1985 > URL: https://issues.apache.org/jira/browse/STORM-1985 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Jungtaek Lim > Labels: newbie > > After STORM-1976, Nimbus doesn't clean up corrupted topologies. > (corrupted topology means the topology whose codes are not available on > blobstore.) > Also after STORM-1977, no Nimbus is gaining leadership if one or more > topologies are corrupted, which means all nimbuses will be no-op. > So we should provide a tool to kill specific topology without accessing > leader nimbus (because there's no leader nimbus at that time). The tool > should also determine which topologies are corrupted, and show its list or > clean up automatically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-2020) Stop using sun internal classes
[ https://issues.apache.org/jira/browse/STORM-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-2020. Resolution: Fixed Fix Version/s: 2.0.0 > Stop using sun internal classes > --- > > Key: STORM-2020 > URL: https://issues.apache.org/jira/browse/STORM-2020 > Project: Apache Storm > Issue Type: Bug > Components: storm-hdfs, storm-redis >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 2.0.0 > > > sun.reflect.generics.reflectiveObjects.NotImplementedException, > sun.misc.BASE64Decoder, and sun.misc.BASE64Encoder are not public APIs we > should not be using them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-2022) FieldsTest.selectingUnknownFieldThrowsTest is failing
[
https://issues.apache.org/jira/browse/STORM-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-2022.
Resolution: Fixed
Fix Version/s: 2.0.0
> FieldsTest.selectingUnknownFieldThrowsTest is failing
> -
>
> Key: STORM-2022
> URL: https://issues.apache.org/jira/browse/STORM-2022
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 2.0.0
>
>
> {code}
> classname="org.apache.storm.tuple.FieldsTest" time="0.007">
> type="java.lang.Exception">
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (STORM-2021) storm-kinesis missing licenses
[
https://issues.apache.org/jira/browse/STORM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-2021.
Resolution: Fixed
Assignee: Robert Joseph Evans (was: Priyank Shah)
Fix Version/s: 2.0.0
> storm-kinesis missing licenses
> --
>
> Key: STORM-2021
> URL: https://issues.apache.org/jira/browse/STORM-2021
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kinesis
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 2.0.0
>
>
> {code}
> Unapproved licenses:
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-2018) Simplify Threading Model of the Supervisor
[ https://issues.apache.org/jira/browse/STORM-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408411#comment-15408411 ] Robert Joseph Evans commented on STORM-2018: Turns out I am going to have to split Waiting For Localization into two states. The first one will wait for the basic things to download, storm.conf, etc. And then another state to read in the conf and wait for the other blobs to download. > Simplify Threading Model of the Supervisor > -- > > Key: STORM-2018 > URL: https://issues.apache.org/jira/browse/STORM-2018 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: Slot.dot, Slot.svg > > > We have been trying to roll out CGROUP enforcement and right now are running > into a number of race conditions in the supervisor. When using CGROUPS the > timing of some operations are different and are exposing issues that we would > not see without this. > In order to make progress with testing/deploying CGROUP and RAS we are going > to try and refactor the supervisor to have a simpler threading model, but > likely with more threads. We will base the code off of the java code > currently in master, and may replace that in the 2.0 release, but plan on > having it be a part of 1.x too, if it truly is more stable. > I will try to keep this JIRA up to date with what we are doing and the > architecture to keep the community informed. We need to move quickly to meet > some of our company goals but will not just shove this in. We welcome any > feedback on the design and code before it goes into the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-2021) storm-kinesis missing licenses
[
https://issues.apache.org/jira/browse/STORM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408048#comment-15408048
]
Robert Joseph Evans commented on STORM-2021:
Actually I'll just do it.
> storm-kinesis missing licenses
> --
>
> Key: STORM-2021
> URL: https://issues.apache.org/jira/browse/STORM-2021
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kinesis
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Priyank Shah
>Priority: Blocker
>
> {code}
> Unapproved licenses:
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (STORM-2021) storm-kinesis missing licenses
[
https://issues.apache.org/jira/browse/STORM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reassigned STORM-2021:
--
Assignee: Robert Joseph Evans (was: Priyank Shah)
> storm-kinesis missing licenses
> --
>
> Key: STORM-2021
> URL: https://issues.apache.org/jira/browse/STORM-2021
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kinesis
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> {code}
> Unapproved licenses:
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (STORM-2022) FieldsTest.selectingUnknownFieldThrowsTest is failing
[
https://issues.apache.org/jira/browse/STORM-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans reassigned STORM-2022:
--
Assignee: Robert Joseph Evans
> FieldsTest.selectingUnknownFieldThrowsTest is failing
> -
>
> Key: STORM-2022
> URL: https://issues.apache.org/jira/browse/STORM-2022
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>
> {code}
> classname="org.apache.storm.tuple.FieldsTest" time="0.007">
> type="java.lang.Exception">
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-2022) FieldsTest.selectingUnknownFieldThrowsTest is failing
[
https://issues.apache.org/jira/browse/STORM-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408029#comment-15408029
]
Robert Joseph Evans commented on STORM-2022:
Looks like the code was updated, but the test was not.
> FieldsTest.selectingUnknownFieldThrowsTest is failing
> -
>
> Key: STORM-2022
> URL: https://issues.apache.org/jira/browse/STORM-2022
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>
> {code}
> classname="org.apache.storm.tuple.FieldsTest" time="0.007">
> type="java.lang.Exception">
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-2022) FieldsTest.selectingUnknownFieldThrowsTest is failing
Robert Joseph Evans created STORM-2022:
--
Summary: FieldsTest.selectingUnknownFieldThrowsTest is failing
Key: STORM-2022
URL: https://issues.apache.org/jira/browse/STORM-2022
Project: Apache Storm
Issue Type: Bug
Components: storm-core
Affects Versions: 2.0.0
Reporter: Robert Joseph Evans
{code}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-2021) storm-kinesis missing licenses
[
https://issues.apache.org/jira/browse/STORM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408014#comment-15408014
]
Robert Joseph Evans commented on STORM-2021:
[~pshah],
I assume that you wanted to have all of the files to be apache licensed.
Please take a look at this it is breaking travis.
> storm-kinesis missing licenses
> --
>
> Key: STORM-2021
> URL: https://issues.apache.org/jira/browse/STORM-2021
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kinesis
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Priority: Blocker
>
> {code}
> Unapproved licenses:
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-2021) storm-kinesis missing licenses
Robert Joseph Evans created STORM-2021:
--
Summary: storm-kinesis missing licenses
Key: STORM-2021
URL: https://issues.apache.org/jira/browse/STORM-2021
Project: Apache Storm
Issue Type: Bug
Components: storm-kinesis
Affects Versions: 2.0.0
Reporter: Robert Joseph Evans
Priority: Blocker
{code}
Unapproved licenses:
external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-2021) storm-kinesis missing licenses
[
https://issues.apache.org/jira/browse/STORM-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated STORM-2021:
---
Assignee: Priyank Shah
> storm-kinesis missing licenses
> --
>
> Key: STORM-2021
> URL: https://issues.apache.org/jira/browse/STORM-2021
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kinesis
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Priyank Shah
>Priority: Blocker
>
> {code}
> Unapproved licenses:
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisBoltTest.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/KinesisSpoutTopology.java
>
> external/storm-kinesis/src/test/java/org/apache/storm/kinesis/spout/test/TestRecordToTupleMapper.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (STORM-2020) Stop using sun internal classes
[ https://issues.apache.org/jira/browse/STORM-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned STORM-2020: -- Assignee: Robert Joseph Evans > Stop using sun internal classes > --- > > Key: STORM-2020 > URL: https://issues.apache.org/jira/browse/STORM-2020 > Project: Apache Storm > Issue Type: Bug > Components: storm-hdfs, storm-redis >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > sun.reflect.generics.reflectiveObjects.NotImplementedException, > sun.misc.BASE64Decoder, and sun.misc.BASE64Encoder are not public APIs we > should not be using them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-2020) Stop using sun internal classes
Robert Joseph Evans created STORM-2020: -- Summary: Stop using sun internal classes Key: STORM-2020 URL: https://issues.apache.org/jira/browse/STORM-2020 Project: Apache Storm Issue Type: Bug Components: storm-hdfs, storm-redis Affects Versions: 2.0.0 Reporter: Robert Joseph Evans sun.reflect.generics.reflectiveObjects.NotImplementedException, sun.misc.BASE64Decoder, and sun.misc.BASE64Encoder are not public APIs we should not be using them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-2015) logviewer does not download file when the directory is a symbolic link fails with 404 page not found
[ https://issues.apache.org/jira/browse/STORM-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406384#comment-15406384 ] Robert Joseph Evans commented on STORM-2015: Actually this is a security issue. We only allow downloading of files that are under a specific known log directory. Otherwise a worker could link to a file that it cannot actually read, but the logviewer can. I think the fix would be to make the a configurable whitelist of allowed subdirectories. > logviewer does not download file when the directory is a symbolic link fails > with 404 page not found > > > Key: STORM-2015 > URL: https://issues.apache.org/jira/browse/STORM-2015 > Project: Apache Storm > Issue Type: Bug >Reporter: saurabh mishra > > logviewer does not download file when the directory is a symbolic link it > fails with 404 page not found. > (defn download-log-file [fname req resp user ^String root-dir] > (let [file (.getCanonicalFile (File. root-dir fname))] > (if (.exists file) > (-> (resp/response "Page not found") > (resp/status 404) > Replace storm root-dir as an actual directory it succeeds to download the > file. > Symbolic link for log locations is standard practice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-2018) Simplify Threading Model of the Supervisor
[
https://issues.apache.org/jira/browse/STORM-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406303#comment-15406303
]
Robert Joseph Evans commented on STORM-2018:
I would propose that we move to a model where we have 3 main types of threads,
for the supervisor itself. Threads in the localizer are different.
1) A Single HB thread (very much like it is now)
2) A Single Scheduling Sync Thread that would
{code}
while (!done) {
read scheduling from ZK && sanity check with retry like today;
for (int port: set.union(scheduling.ports, slots.keys)) {
Slot s = slots.get(port);
if (s == null) {
s = new Slot();
slots.put(port, s);
}
s.setNewAssignment(scheduling.get(port));
}
sleep(...);
}
{code}
3) A Slot thread per slot. This thread would more or less do the following
{code}
while(!done) {
Assignment newAssignment = this.newAssignment;
StateMachine.transitionIfNeeded(newAssignment,...);
}
{code}
The state machine itself is described in
[Slot.dot|https://issues.apache.org/jira/secure/attachment/12821873/Slot.dot]
and you can see a visualization in Slot.svg
!Slot.svg!
Slot would have just a few methods to set things asynchronously
{code}
public void setNewAssignment(Assignment...);
public void informWorkerDied(String workerId...);
{code}
Every time that current assignment is written to it would also be written out
to disk so if we crash we can recover.
> Simplify Threading Model of the Supervisor
> --
>
> Key: STORM-2018
> URL: https://issues.apache.org/jira/browse/STORM-2018
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: Slot.dot, Slot.svg
>
>
> We have been trying to roll out CGROUP enforcement and right now are running
> into a number of race conditions in the supervisor. When using CGROUPS the
> timing of some operations are different and are exposing issues that we would
> not see without this.
> In order to make progress with testing/deploying CGROUP and RAS we are going
> to try and refactor the supervisor to have a simpler threading model, but
> likely with more threads. We will base the code off of the java code
> currently in master, and may replace that in the 2.0 release, but plan on
> having it be a part of 1.x too, if it truly is more stable.
> I will try to keep this JIRA up to date with what we are doing and the
> architecture to keep the community informed. We need to move quickly to meet
> some of our company goals but will not just shove this in. We welcome any
> feedback on the design and code before it goes into the community.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-2018) Simplify Threading Model of the Supervisor
[ https://issues.apache.org/jira/browse/STORM-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-2018: --- Attachment: Slot.svg Slot.dot > Simplify Threading Model of the Supervisor > -- > > Key: STORM-2018 > URL: https://issues.apache.org/jira/browse/STORM-2018 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: Slot.dot, Slot.svg > > > We have been trying to roll out CGROUP enforcement and right now are running > into a number of race conditions in the supervisor. When using CGROUPS the > timing of some operations are different and are exposing issues that we would > not see without this. > In order to make progress with testing/deploying CGROUP and RAS we are going > to try and refactor the supervisor to have a simpler threading model, but > likely with more threads. We will base the code off of the java code > currently in master, and may replace that in the 2.0 release, but plan on > having it be a part of 1.x too, if it truly is more stable. > I will try to keep this JIRA up to date with what we are doing and the > architecture to keep the community informed. We need to move quickly to meet > some of our company goals but will not just shove this in. We welcome any > feedback on the design and code before it goes into the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-2018) Simplify Threading Model of the Supervisor
Robert Joseph Evans created STORM-2018: -- Summary: Simplify Threading Model of the Supervisor Key: STORM-2018 URL: https://issues.apache.org/jira/browse/STORM-2018 Project: Apache Storm Issue Type: New Feature Components: storm-core Affects Versions: 1.0.0, 2.0.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans We have been trying to roll out CGROUP enforcement and right now are running into a number of race conditions in the supervisor. When using CGROUPS the timing of some operations are different and are exposing issues that we would not see without this. In order to make progress with testing/deploying CGROUP and RAS we are going to try and refactor the supervisor to have a simpler threading model, but likely with more threads. We will base the code off of the java code currently in master, and may replace that in the 2.0 release, but plan on having it be a part of 1.x too, if it truly is more stable. I will try to keep this JIRA up to date with what we are doing and the architecture to keep the community informed. We need to move quickly to meet some of our company goals but will not just shove this in. We welcome any feedback on the design and code before it goes into the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1981) storm command hangs
[
https://issues.apache.org/jira/browse/STORM-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384577#comment-15384577
]
Robert Joseph Evans commented on STORM-1981:
OK Makes since thanks for clarifying [~raghavgautam]
> storm command hangs
> ---
>
> Key: STORM-1981
> URL: https://issues.apache.org/jira/browse/STORM-1981
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>
> I ran the following command which threw an exception and never completed. I
> understand that the topology has some issues which caused this hang. The
> problem that I want to emphasize is the *hang*. We can't assume that user
> code will always be doing things correctly.
> {code}
> test@host:/grid/0/hadoopqe$ /usr/hdp/current/storm-client/bin/storm -c
> java.security.auth.login.config=/etc/storm/conf/client_jaas.conf -c
> storm.thrift.transport=org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
> jar
> /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-1.0.1.jar
> org.apache.storm.starter.trident.TridentKafkaWordCount
> nat-d7-vnas-storm-5.openstacklocal:6667
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -server -Ddaemon.name=
> -Dstorm.options=java.security.auth.login.config%3D%2Fetc%2Fstorm%2Fconf%2Fclient_jaas.conf,storm.thrift.transport%3Dorg.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
> -Dstorm.home=/storm -Dstorm.log.dir=/grid/0/log/storm
> -Djava.library.path=/usr/local/lib:/o\
> pt/local/lib:/usr/lib -Dstorm.conf.file= -cp
> /storm/lib/slf4j-api-1.7.7.jar:/storm/lib/clojure-1.7.0.jar:/storm/lib/minlog-1.3.0.jar:/storm/lib/log4j-over-slf4j-1.6.6.jar:/storm/lib/ambari-metrics-storm-sink.jar:/storm/lib/reflectasm-1.10.1.jar:/storm/lib/ring-cors-0.1.5.jar:/storm/lib/disruptor-3.3.2.jar:/storm/lib/storm-rename-hack-1.0.1.jar:/storm/lib/log4j-api-2.1.jar:/storm/lib/log4j-slf4j-impl-2.1.jar:/storm/lib/zoo\
> keeper.jar:/storm/lib/kryo-3.0.3.jar:/storm/lib/log4j-core-2.1.jar:/storm/lib/storm-core-1.0.1.jar:/storm/lib/servlet-api-2.5.jar:/storm/lib/objenesis-2.1.jar:/storm/lib/asm-5.0.3.jar:/storm/extlib/scala-library-2.10.4.jar:/storm/extlib/storm-kafka-1.0.1.jar:/storm/extlib/kafka_2.10-0.10.0.jar
> org.apache.storm.daemon.ClientJarTransformerRunner
> org.apache.storm.hack.StormShadeTransformer
> /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-1.0.1.jar
> /tmp/1bb42b0a4d2f11e6871dfa163e434686.jar
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name=
> -Dstorm.options=java.security.auth.login.config%3D%2Fetc%2Fstorm%2Fconf%2Fclient_jaas.conf,storm.thrift.transport%3Dorg.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
> -Dstorm.home=/storm -Dstorm.log.dir=/grid/0/log/storm
> -Djava.library.path=/usr/local/lib:/o\
> pt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file=
> -cp
> /storm/lib/slf4j-api-1.7.7.jar:/storm/lib/clojure-1.7.0.jar:/storm/lib/minlog-1.3.0.jar:/storm/lib/log4j-over-slf4j-1.6.6.jar:/storm/lib/ambari-metrics-storm-sink.jar:/storm/lib/reflectasm-1.10.1.jar:/storm/lib/ring-cors-0.1.5.jar:/storm/lib/disruptor-3.3.2.jar:/storm/lib/storm-rename-hack-1.0.1.jar:/storm/lib/log4j-api-2.1.jar:/storm/lib/log4j-slf4j-impl-2.1.jar:/gri\
> d/0/hdp/2.5.0.0-1016/storm/lib/zookeeper.jar:/storm/lib/kryo-3.0.3.jar:/storm/lib/log4j-core-2.1.jar:/storm/lib/storm-core-1.0.1.jar:/storm/lib/servlet-api-2.5.jar:/storm/lib/objenesis-2.1.jar:/storm/lib/asm-5.0.3.jar:/storm/extlib/scala-library-2.10.4.jar:/storm/extlib/storm-kafka-1.0.1.jar:/storm/extlib/kafka_2.10-0.10.0.jar:/tmp/1bb42b0a4d2f11e6871dfa163e434686.jar:/usr/hdp/current/storm-supervisor/conf:/storm/bin
> -Dstorm.jar=/tmp/1bb42b0a4d2f11e6871dfa163e434686.jar
> org.apache.storm.starter.trident.TridentKafkaWordCount
> nat-d7-vnas-storm-5.openstacklocal:6667
> Using Kafka zookeeper url: nat-d7-vnas-storm-5.openstacklocal:6667 broker
> url: localhost:9092
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at clojure.lang.RT.classForName(RT.java:2154)
> at clojure.lang.RT.classForName(RT.java:2163)
> at clojure.lang.RT.loadClassForName(RT.java:2182)
> at clojure.lang.RT.load(RT.java:436)
> at clojure.lang.RT.load(RT.java:412)
> at clojure.core$load$fn__5448.invoke(core.clj:5866)
> at clojure.core$load.doInvoke(core.clj:5865)
> ...
> ...
> ...
> at clojure.lang.Var.invoke(Var.java:379)
> at org.apache.storm.LocalCluster.(Unknown Source)
> at
> org.apache.storm.starter.trident.TridentKafkaWordCount.runMain(TridentKafkaWordCount.java:254)
> at
> org.apache.storm.starter.trident.TridentKafkaWordCount.main(TridentKafkaWordCount.java:240)
[jira] [Commented] (STORM-1981) storm command hangs
[
https://issues.apache.org/jira/browse/STORM-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384184#comment-15384184
]
Robert Joseph Evans commented on STORM-1981:
The code indicates that your command is looking to pull in
{{org.apache.ranger.authorization.storm.authorizer.RangerStormAuthorizer}} but
could not find it. This is not a part of storm, it appears to be a part of
ranger, so my guess would be that you need to include ranger, or part of it on
your classpath. You could do this by putting it in the lib directory under the
storm.home directory. You might also want to try asking on the ranger or storm
user mailing lists, because honestly I don't use ranger. At this point it does
not look like a bug in storm, it looks like user error or an issue with your
configuration, so I am going to close this JIRA. If you disagree that it is
actually a bug in storm feel free to reopen this.
> storm command hangs
> ---
>
> Key: STORM-1981
> URL: https://issues.apache.org/jira/browse/STORM-1981
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>
> I ran the following command which threw an exception and never completed. I
> understand that the topology has some issues which caused this hang. The
> problem that I want to emphasize is the *hang*. We can't assume that user
> code will always be doing things correctly.
> {code}
> test@host:/grid/0/hadoopqe$ /usr/hdp/current/storm-client/bin/storm -c
> java.security.auth.login.config=/etc/storm/conf/client_jaas.conf -c
> storm.thrift.transport=org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
> jar
> /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-1.0.1.jar
> org.apache.storm.starter.trident.TridentKafkaWordCount
> nat-d7-vnas-storm-5.openstacklocal:6667
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -server -Ddaemon.name=
> -Dstorm.options=java.security.auth.login.config%3D%2Fetc%2Fstorm%2Fconf%2Fclient_jaas.conf,storm.thrift.transport%3Dorg.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
> -Dstorm.home=/storm -Dstorm.log.dir=/grid/0/log/storm
> -Djava.library.path=/usr/local/lib:/o\
> pt/local/lib:/usr/lib -Dstorm.conf.file= -cp
> /storm/lib/slf4j-api-1.7.7.jar:/storm/lib/clojure-1.7.0.jar:/storm/lib/minlog-1.3.0.jar:/storm/lib/log4j-over-slf4j-1.6.6.jar:/storm/lib/ambari-metrics-storm-sink.jar:/storm/lib/reflectasm-1.10.1.jar:/storm/lib/ring-cors-0.1.5.jar:/storm/lib/disruptor-3.3.2.jar:/storm/lib/storm-rename-hack-1.0.1.jar:/storm/lib/log4j-api-2.1.jar:/storm/lib/log4j-slf4j-impl-2.1.jar:/storm/lib/zoo\
> keeper.jar:/storm/lib/kryo-3.0.3.jar:/storm/lib/log4j-core-2.1.jar:/storm/lib/storm-core-1.0.1.jar:/storm/lib/servlet-api-2.5.jar:/storm/lib/objenesis-2.1.jar:/storm/lib/asm-5.0.3.jar:/storm/extlib/scala-library-2.10.4.jar:/storm/extlib/storm-kafka-1.0.1.jar:/storm/extlib/kafka_2.10-0.10.0.jar
> org.apache.storm.daemon.ClientJarTransformerRunner
> org.apache.storm.hack.StormShadeTransformer
> /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-1.0.1.jar
> /tmp/1bb42b0a4d2f11e6871dfa163e434686.jar
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name=
> -Dstorm.options=java.security.auth.login.config%3D%2Fetc%2Fstorm%2Fconf%2Fclient_jaas.conf,storm.thrift.transport%3Dorg.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin
> -Dstorm.home=/storm -Dstorm.log.dir=/grid/0/log/storm
> -Djava.library.path=/usr/local/lib:/o\
> pt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib -Dstorm.conf.file=
> -cp
> /storm/lib/slf4j-api-1.7.7.jar:/storm/lib/clojure-1.7.0.jar:/storm/lib/minlog-1.3.0.jar:/storm/lib/log4j-over-slf4j-1.6.6.jar:/storm/lib/ambari-metrics-storm-sink.jar:/storm/lib/reflectasm-1.10.1.jar:/storm/lib/ring-cors-0.1.5.jar:/storm/lib/disruptor-3.3.2.jar:/storm/lib/storm-rename-hack-1.0.1.jar:/storm/lib/log4j-api-2.1.jar:/storm/lib/log4j-slf4j-impl-2.1.jar:/gri\
> d/0/hdp/2.5.0.0-1016/storm/lib/zookeeper.jar:/storm/lib/kryo-3.0.3.jar:/storm/lib/log4j-core-2.1.jar:/storm/lib/storm-core-1.0.1.jar:/storm/lib/servlet-api-2.5.jar:/storm/lib/objenesis-2.1.jar:/storm/lib/asm-5.0.3.jar:/storm/extlib/scala-library-2.10.4.jar:/storm/extlib/storm-kafka-1.0.1.jar:/storm/extlib/kafka_2.10-0.10.0.jar:/tmp/1bb42b0a4d2f11e6871dfa163e434686.jar:/usr/hdp/current/storm-supervisor/conf:/storm/bin
> -Dstorm.jar=/tmp/1bb42b0a4d2f11e6871dfa163e434686.jar
> org.apache.storm.starter.trident.TridentKafkaWordCount
> nat-d7-vnas-storm-5.openstacklocal:6667
> Using Kafka zookeeper url: nat-d7-vnas-storm-5.openstacklocal:6667 broker
> url: localhost:9092
> Exception in thread "main" java.lang.ExceptionInInitializerError
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
[jira] [Commented] (STORM-1976) Storm Nimbus H/A has issue on cleaning corrupted topologies
[
https://issues.apache.org/jira/browse/STORM-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382351#comment-15382351
]
Robert Joseph Evans commented on STORM-1976:
For me I keep seeing this. Min replication count was not set but the cluster
was run in HA mode and something bad happened, and then HA did not recover
cleanly. This issue is that without with min replication count HA does not
really work properly. It would be nice to have a way to make this simpler for
end users so they don't have something else to configure. But the reality is
that a lot of this feels like user errors and errors in documentation, not
blockers for the software. Additionally I don't think we want to "fix" min
replication count not being set quickly. I think it is a more complex problem.
> Storm Nimbus H/A has issue on cleaning corrupted topologies
> ---
>
> Key: STORM-1976
> URL: https://issues.apache.org/jira/browse/STORM-1976
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Jungtaek Lim
>Priority: Blocker
>
> In the following scenario storm-ha runs into issues:
> 1. Kill a non-leader nimbus
> 2. Submit a topology
> 3. Bring up the non-leader nimbus
> After step-3 expectation is that the non-leader nimbus will download topology
> jar. Instead it cleans up the topology.
> {code}
> 2016-07-12 07:11:09.511 o.a.s.c.zookeeper-state-factory [WARN] Received event
> ::none: with disconnected Reader Zookeeper.
> 2016-07-12 07:11:09.587 o.a.s.zookeeper [INFO] Queued up for leader lock.
> 2016-07-12 07:11:09.608 o.a.s.d.nimbus [INFO] Corrupt topology
> JoinedNonLeaderNimbusTriesToDownloadTopologyCode-2-1468307239 has state on
> zookeeper but doesn't have a local dir on Nimbus. Cleaning up...
> 2016-07-12 07:11:09.932 o.a.h.m.s.s.StormTimelineMetricsReporter [INFO]
> Preparing Storm Metrics Reporter
> 2016-07-12 07:11:09.946 o.a.s.d.m.MetricsUtils [INFO] Using statistics
> reporter
> plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1977) Leader Nimbus crashes with getClusterInfo when it doesn't have one or more replicated topology codes
[
https://issues.apache.org/jira/browse/STORM-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382316#comment-15382316
]
Robert Joseph Evans commented on STORM-1977:
The entire procedure to reproduce the issue is part of the problem. If you
don't have both nimbus instances up at the same time they cannot possibly stay
in sync with each other.
> Leader Nimbus crashes with getClusterInfo when it doesn't have one or more
> replicated topology codes
>
>
> Key: STORM-1977
> URL: https://issues.apache.org/jira/browse/STORM-1977
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 1.0.1
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> While investigating STORM-1976, I found that there're cases for nimbus to not
> having topology codes.
> Before BlobStore, only nimbuses which is having all topology codes can gain
> leadership, otherwise they give up leadership immediately. While introducing
> BlobStore, this logic is removed.
> I don't know it's intended or not, but it incurs one of nimbus to gain
> leadership which doesn't have replicated topology code, and the nimbus will
> be crashed when getClusterInfo is requested.
> Easiest way to reproduce is:
> 1. comment cleanup-corrupt-topologies! from nimbus.clj (It's a quick
> workaround for resolving STORM-1976), and patch Storm cluster
> 2. Launch Nimbus 1 (leader)
> 3. Run topology
> 4. Kill Nimbus 1
> 5. Launch Nimbus 2 from different node
> 6. Nimbus 2 gains leadership
> 7. getClusterInfo is requested to Nimbus 2, and Nimbus 2 gets crashed
> Log:
> {code}
> 2016-07-17 08:47:48.378 o.a.s.b.FileBlobStoreImpl [INFO] Creating new blob
> store based in /grid/0/hadoop/storm/blobs
> ...
> 2016-07-17 08:47:48.619 o.a.s.zookeeper [INFO] Queued up for leader lock.
> 2016-07-17 08:47:48.651 o.a.s.zookeeper [INFO] gained leadership
> ...
> 2016-07-17 08:47:48.833 o.a.s.d.nimbus [INFO] Starting nimbus server for
> storm version '1.1.1-SNAPSHOT'
> 2016-07-17 08:47:49.295 o.a.s.t.ProcessFunction [ERROR] Internal error
> processing getClusterInfo
> KeyNotFoundException(msg:production-topology-2-1468745167-stormcode.ser)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:149)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlobReplication(LocalFsBlobStore.java:268)
> ...
> at
> org.apache.storm.daemon.nimbus$get_blob_replication_count.invoke(nimbus.clj:498)
> at
> org.apache.storm.daemon.nimbus$get_cluster_info$iter__9520__9524$fn__9525.invoke(nimbus.clj:1427)
> ...
> at
> org.apache.storm.daemon.nimbus$get_cluster_info.invoke(nimbus.clj:1401)
> at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__9612.getClusterInfo(nimbus.clj:1838)
> at
> org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724)
> at
> org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708)
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ...
> 2016-07-17 08:47:49.397 o.a.s.b.BlobStoreUtils [ERROR] Could not download
> blob with keyproduction-topology-2-1468745167-stormconf.ser
> 2016-07-17 08:47:49.400 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyproduction-topology-2-1468745167-stormconf.ser
> 2016-07-17 08:47:49.402 o.a.s.d.nimbus [ERROR] Error when processing event
> KeyNotFoundException(msg:production-topology-2-1468745167-stormconf.ser)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:149)
> at
> org.apache.storm.blobstore.LocalFsBlobStore.getBlob(LocalFsBlobStore.java:239)
> at org.apache.storm.blobstore.BlobStore.readBlobTo(BlobStore.java:271)
> at org.apache.storm.blobstore.BlobStore.readBlob(BlobStore.java:300)
> ...
>at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> at
> org.apache.storm.daemon.nimbus$read_storm_conf_as_nimbus.invoke(nimbus.clj:548)
> at
> org.apache.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:555)
> at
> org.apache.storm.daemon.nimbus$mk_assignments$iter__9205__9209$fn__9210.invoke(nimbus.clj:912)
> ...
> at
> org.apache.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:911)
> at clojure.lang.RestFn.invoke(RestFn.java:410)
> at
> org.apache.storm.daemon.nimbus$fn__9769$exec_fn__1363__auto9770$fn__9781$fn__9782.invoke(nimbus.clj:2216)
> at
>
[jira] [Closed] (STORM-1959) KafkaPartitionOffsetLag.java does not have license
[ https://issues.apache.org/jira/browse/STORM-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans closed STORM-1959. -- Resolution: Fixed Fix Version/s: 1.1.0 2.0.0 Thanks [~kabhwan] I merged this into master and 1.x-branch. Thanks again for the quick turn around time on this. > KafkaPartitionOffsetLag.java does not have license > -- > > Key: STORM-1959 > URL: https://issues.apache.org/jira/browse/STORM-1959 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-monitor >Affects Versions: 2.0.0, 1.1.0 >Reporter: Robert Joseph Evans >Assignee: Jungtaek Lim >Priority: Blocker > Fix For: 2.0.0, 1.1.0 > > > RAT is failing > external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaPartitionOffsetLag.java > Looks like this was introduced as a part of STORM-1950 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default
[ https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370999#comment-15370999 ] Robert Joseph Evans commented on STORM-1096: [~harsha_ch], Then it sounds like we are trying to enforce security when it is turned off, and that is the real underlying problem. > UI tries to impersonate wrong user when getting topology conf for > authorization, impersonation is allowed by default > > > Key: STORM-1096 > URL: https://issues.apache.org/jira/browse/STORM-1096 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.10.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.10.0 > > > We have started using 0.10.0 under load and found a few issues around the UI > and impersonation. > The UI when trying to connect to nimbus will impersonate other users. > Nimbus, by default allows impersonation and just outputs a warning message > that it is allowed. We really should default to not allowing impersonation. > having the authorizer configured by default does not hurt when running > insecure because impersonation is not possible, but when security is enabled > if someone forgets to set this config we are now insecure by default. > If you do set all of that up correctly the UI now can impersonate the wrong > user when connecting to nimbus. > The UI decides which user to impersonate by pulling it from the request > context. The requestContext is populated from the HttpRequest when > assert-authorized-user is called. assert-authorized-user takes a > topology-conf as a parameter. The only way to get this topology conf is to > talk to nimbus, which will get the wrong user because the request context has > not been populated yet. > This just because a huge pain for users who way too often will not be able to > see pages on the UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1950) Change response json of "Topology Lag" REST API to keyed by spoutId, topic, partition
[
https://issues.apache.org/jira/browse/STORM-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370924#comment-15370924
]
Robert Joseph Evans commented on STORM-1950:
[~sriharsha] and [~kabhwan],
Could you please take a look at STORM-1959. It is causing the travis builds to
fail.
> Change response json of "Topology Lag" REST API to keyed by spoutId, topic,
> partition
> -
>
> Key: STORM-1950
> URL: https://issues.apache.org/jira/browse/STORM-1950
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-ui
>Affects Versions: 1.1.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
> Fix For: 2.0.0, 1.1.0
>
>
> From code review for STORM-1945, there's an idea to change JSON response of
> "Topology Lag" API to keyed by topic, partition number.
> https://github.com/apache/storm/pull/1541#issuecomment-230983140
> I think also make result keyed by spout id would be good.
> Here's sample JSON of output after this issue is resolved.
> {code}
> {
>"spout1":{
> "spoutId":"spout1",
> "spoutType":"KAFKA",
> "spoutLagResult":{
> "topic":{
> "partition0":{
>"consumerCommittedOffset":1175610,
>"logHeadOffset":5634192,
>"lag":4458582
> },
> "partition2":{
>"consumerCommittedOffset":1175610,
>"logHeadOffset":5634192,
>"lag":4458582
> }
> },
> "topic2":{
> "partition0":{
>"consumerCommittedOffset":1175610,
>"logHeadOffset":5634192,
>"lag":4458582
> },
> "partition2":{
>"consumerCommittedOffset":1175610,
>"logHeadOffset":5634192,
>"lag":4458582
> }
> }
> }
>}
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-1959) KafkaPartitionOffsetLag.java does not have license
Robert Joseph Evans created STORM-1959: -- Summary: KafkaPartitionOffsetLag.java does not have license Key: STORM-1959 URL: https://issues.apache.org/jira/browse/STORM-1959 Project: Apache Storm Issue Type: Bug Components: storm-kafka-monitor Affects Versions: 2.0.0 Reporter: Robert Joseph Evans Priority: Blocker RAT is failing external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaPartitionOffsetLag.java Looks like this was introduced as a part of STORM-1950 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology
[ https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370729#comment-15370729 ] Robert Joseph Evans commented on STORM-1949: I am not sure that we can use our internal messaging as is to make this work. The messages to turn on and off backpressure would likely get blocked behind actual data messages and never make it to the spout that is throttled off. I would prefer to see us go towards the model of JStorm and have a topology master, there is a lot of work that ZK is doing that could be moved over to a master. It could easily take over the responsibilities of both pacemaker/ZK(for metrics) and ZK for backpressure. I'm not sure I am ready to let the master reschedule the topology itself, I prefer to still have centralized scheduling, but I could be persuaded. > Backpressure can cause spout to stop emitting and stall topology > > > Key: STORM-1949 > URL: https://issues.apache.org/jira/browse/STORM-1949 > Project: Apache Storm > Issue Type: Bug >Reporter: Roshan Naik > > Problem can be reproduced by this [Word count > topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java] > within a IDE. > I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt > instances. > The problem is more easily reproduced with WC topology as it causes an > explosion of tuples due to splitting a sentence tuple into word tuples. As > the bolts have to process more tuples than the spout is producing, spout > needs to operate slower. > The amount of time it takes for the topology to stall can vary.. but > typically under 10 mins. > *My theory:* I suspect there is a race condition in the way ZK is being > utilized to enable/disable back pressure. When congested (i.e pressure > exceeds high water mark), the bolt's worker records this congested situation > in ZK by creating a node. Once the congestion is reduced below the low water > mark, it deletes this node. > The spout's worker has setup a watch on the parent node, expecting a callback > whenever there is change in the child nodes. On receiving the callback the > spout's worker lists the parent node to check if there are 0 or more child > nodes it is essentially trying to figure out the nature of state change > in ZK to determine whether to throttle or not. Subsequently it setsup > another watch in ZK to keep an eye on future changes. > When there are multiple bolts, there can be rapid creation/deletion of these > ZK nodes. Between the time the worker receives a callback and sets up the > next watch.. many changes may have undergone in ZK which will go unnoticed by > the spout. > The condition that the bolts are no longer congested may not get noticed as a > result of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1952) Keeping topology code for supervisor until topology got killed
[ https://issues.apache.org/jira/browse/STORM-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370716#comment-15370716 ] Robert Joseph Evans commented on STORM-1952: There are two situations here. 1) The topology is still up and running, and for some reason it got rescheduled some place else. 2) The topology is killed and will not ever come back. This JIRA is trying to address the first one because when it happens it can take a long time for the topology to come back up on a node that it was running on previously (think about a rolling upgrade). This is why I suggested using the dist cache code with LRU, because it will keep the jar/etc around longer, but not forever, if the disk space is needed for something else. [~kabhwan], I think your comment is about the second situation and if so yes you are correct about that too. If we do modify the topology specific download to use the cache code it would be good to add an optimization to remove them from the cache when we know it will never be used again. But that I see as an optimization, not necessarily as a requirement. > Keeping topology code for supervisor until topology got killed > -- > > Key: STORM-1952 > URL: https://issues.apache.org/jira/browse/STORM-1952 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 1.0.0, 2.0.0, 1.0.1 >Reporter: Jungtaek Lim > > It's based on review comment from [~sriharsha]. > https://github.com/apache/storm/pull/1528/files#r69152524 > Please feel free to change reporter if you would like to. > In supervisor we're removing topology code when assignments for that > supervisor has gone. > But there's valid scenario to need to keep the topology code though > assignments for that supervisor is none, for example, rebalancing. > So it would be better for supervisor to keep topology code until topology has > been killed (and all topology workers assigned to that supervisor are also > killed). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default
[
https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368572#comment-15368572
]
Robert Joseph Evans commented on STORM-1096:
realPrincipal is the actual user making the thrift call. In the case of a
normal UI interaction/REST call this is the UI user.
principal is the user being impersonated by the thrift call. In the case of a
normal UI interaction/REST call this is the user that authenticated with the UI
("topology-user1").
>From the ImpersonationAuthorizer code we see that the key is the super user.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/security/auth/authorizer/ImpersonationAuthorizer.java#L65-66
https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/security/auth/authorizer/ImpersonationAuthorizer.java#L76
Now the next question is what happens if I try to double impersonate. If I
want to do a REST call to the UI impersonating another user.
{{?doAsUser=testUSer1}}
The DefaultHttpCredentailsPlugin will set up the impersonation
https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/security/auth/DefaultHttpCredentialsPlugin.java#L74-L84
inside core we are checking it with ImpersonationAuthorizer, and then we pull
the user out (not the real user but the user being impersonated, and pass it on
to be used when the UI tries to impersonate a user)
https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/ui/core.clj#L87-L112
https://github.com/apache/storm/blob/master/storm-core/src/clj/org/apache/storm/ui/core.clj#L1063-L1065
https://github.com/apache/storm/blob/master/storm-clojure/src/clj/org/apache/storm/thrift.clj#L94
To me it looks like everything is already written how you want it to be. There
could be some bugs in there because I honestly don't use the double
impersonation, but it should be working how you want it to.
> UI tries to impersonate wrong user when getting topology conf for
> authorization, impersonation is allowed by default
>
>
> Key: STORM-1096
> URL: https://issues.apache.org/jira/browse/STORM-1096
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.10.0
>
>
> We have started using 0.10.0 under load and found a few issues around the UI
> and impersonation.
> The UI when trying to connect to nimbus will impersonate other users.
> Nimbus, by default allows impersonation and just outputs a warning message
> that it is allowed. We really should default to not allowing impersonation.
> having the authorizer configured by default does not hurt when running
> insecure because impersonation is not possible, but when security is enabled
> if someone forgets to set this config we are now insecure by default.
> If you do set all of that up correctly the UI now can impersonate the wrong
> user when connecting to nimbus.
> The UI decides which user to impersonate by pulling it from the request
> context. The requestContext is populated from the HttpRequest when
> assert-authorized-user is called. assert-authorized-user takes a
> topology-conf as a parameter. The only way to get this topology conf is to
> talk to nimbus, which will get the wrong user because the request context has
> not been populated yet.
> This just because a huge pain for users who way too often will not be able to
> see pages on the UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology
[ https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368525#comment-15368525 ] Robert Joseph Evans commented on STORM-1949: [~roshan_naik], When writing to ZK every 100 ms we will check to be sure that the ZK state matches the local worker state, just in case there is a race condition and they got off some how. So yes we should have a backup thread to poll ZK for backpressure in case we get off some how with the triggers. As for the load on ZooKeeper yes, that too is one of my concerns, and we at Yahoo have been holding off turning it on by default until it has stabilized a bit and until we have moved all of our clusters to pacemaker. We expect to turn it on fairly soon. How expensive is it though? If you are running zookeeper in the default configuration it can be very expensive because you are constrained by a single disk's IOps, and if you have lots of throttling going on then those IOps can get eaten up fairly quickly. If you have the flush turned off, then it is going to be constrained by the write log to the disk, or by the network whichever saturates first. If it is the write log then you get about 80MB/sec of writes, this is just writing a small amount of metadata, so it should not have that big of an impact, but I have not really measured it. If the network then reads will likely saturate first because of the listings, but again this is a relatively small amount of metadata, so it should have a relatively small impact. If you want to turn it off in defaults.yaml that is fine, just file a separate JIRA for that. If you have a better solution that will reduce the load on ZK that would be great, again file another JIRA for that and we can discuss details there. I am skeptical that we will get anywhere near as good of a throughput if we put in a delay of some sort, and I think we would start to run into issues with heaps filling up if we put in too much of a delay. > Backpressure can cause spout to stop emitting and stall topology > > > Key: STORM-1949 > URL: https://issues.apache.org/jira/browse/STORM-1949 > Project: Apache Storm > Issue Type: Bug >Reporter: Roshan Naik > > Problem can be reproduced by this [Word count > topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java] > within a IDE. > I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt > instances. > The problem is more easily reproduced with WC topology as it causes an > explosion of tuples due to splitting a sentence tuple into word tuples. As > the bolts have to process more tuples than the spout is producing, spout > needs to operate slower. > The amount of time it takes for the topology to stall can vary.. but > typically under 10 mins. > *My theory:* I suspect there is a race condition in the way ZK is being > utilized to enable/disable back pressure. When congested (i.e pressure > exceeds high water mark), the bolt's worker records this congested situation > in ZK by creating a node. Once the congestion is reduced below the low water > mark, it deletes this node. > The spout's worker has setup a watch on the parent node, expecting a callback > whenever there is change in the child nodes. On receiving the callback the > spout's worker lists the parent node to check if there are 0 or more child > nodes it is essentially trying to figure out the nature of state change > in ZK to determine whether to throttle or not. Subsequently it setsup > another watch in ZK to keep an eye on future changes. > When there are multiple bolts, there can be rapid creation/deletion of these > ZK nodes. Between the time the worker receives a callback and sets up the > next watch.. many changes may have undergone in ZK which will go unnoticed by > the spout. > The condition that the bolts are no longer congested may not get noticed as a > result of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1096) UI tries to impersonate wrong user when getting topology conf for authorization, impersonation is allowed by default
[
https://issues.apache.org/jira/browse/STORM-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368287#comment-15368287
]
Robert Joseph Evans commented on STORM-1096:
[~sriharsha]
Sorry I have been out a lot lately and I am now just catching up.
There were a few things that went into this one pull request.
1) {{nimbus.impersonation.authorizer}} was set to
{{"backtype.storm.security.auth.authorizer.ImpersonationAuthorizer"}} by
default. This was done so when security is enabled it is one less config that
needs to be changed when enabling security to get it to work, and to fail
closed (like you mentioned)
2) {{populate-context!}} was added to make sure that the user we impersonate is
always the one from the UI.
3) In the servlet callback handler we null out the "user" to try and avoid
these sort of leaks again in the future.
First of all to clarify a few things. Having impersonation locked down should
not prevent someone from submitting a topology. It prevents someone from
submitting a topology as someone else (which is the entire point of this JIRA).
They can still submit a topology as themselves, without impersonation. If that
does not work then we need to file a separate JIRA and fix that, because I
should always be able to submit something as myself no ACL required.
I am confused about what behavior is the opposite of what we want.
>From what I see if you want ambari and the UI to both work, you would set
>{{nimbus.impersonation.acl}} to something like
{code}
nimbus.impersonation.acl:
ambari-server-storm: // super user
users: [*]
hosts: [*]
ui-storm: //super user
users: [*]
hosts: ["my.nimbus.host"]
{code}
I don't really see why this is hard to automate. If you have a cleaner way to
do it I am all in favor of it, but we cannot default to allowing anyone to
impersonate anyone else by default. It is too easy for someone to shoot
themselves in the foot and be completely insecure when they think thy are OK.
> UI tries to impersonate wrong user when getting topology conf for
> authorization, impersonation is allowed by default
>
>
> Key: STORM-1096
> URL: https://issues.apache.org/jira/browse/STORM-1096
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 0.10.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.10.0
>
>
> We have started using 0.10.0 under load and found a few issues around the UI
> and impersonation.
> The UI when trying to connect to nimbus will impersonate other users.
> Nimbus, by default allows impersonation and just outputs a warning message
> that it is allowed. We really should default to not allowing impersonation.
> having the authorizer configured by default does not hurt when running
> insecure because impersonation is not possible, but when security is enabled
> if someone forgets to set this config we are now insecure by default.
> If you do set all of that up correctly the UI now can impersonate the wrong
> user when connecting to nimbus.
> The UI decides which user to impersonate by pulling it from the request
> context. The requestContext is populated from the HttpRequest when
> assert-authorized-user is called. assert-authorized-user takes a
> topology-conf as a parameter. The only way to get this topology conf is to
> talk to nimbus, which will get the wrong user because the request context has
> not been populated yet.
> This just because a huge pain for users who way too often will not be able to
> see pages on the UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1949) Backpressure can cause spout to stop emitting and stall topology
[ https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367799#comment-15367799 ] Robert Joseph Evans commented on STORM-1949: We have a backup on the write where we will poll every 100ms to be sure everything is in sync. We should have a backup polling for reading the data from ZK too. I thought we did, because we have that on all of the other triggers from ZK, but looking at the code I don't see it. > Backpressure can cause spout to stop emitting and stall topology > > > Key: STORM-1949 > URL: https://issues.apache.org/jira/browse/STORM-1949 > Project: Apache Storm > Issue Type: Bug >Reporter: Roshan Naik > > Problem can be reproduced by this [Word count > topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java] > within a IDE. > I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt > instances. > The problem is more easily reproduced with WC topology as it causes an > explosion of tuples due to splitting a sentence tuple into word tuples. As > the bolts have to process more tuples than the spout is producing, spout > needs to operate slower. > The amount of time it takes for the topology to stall can vary.. but > typically under 10 mins. > *My theory:* I suspect there is a race condition in the way ZK is being > utilized to enable/disable back pressure. When congested (i.e pressure > exceeds high water mark), the bolt's worker records this congested situation > in ZK by creating a node. Once the congestion is reduced below the low water > mark, it deletes this node. > The spout's worker has setup a watch on the parent node, expecting a callback > whenever there is change in the child nodes. On receiving the callback the > spout's worker lists the parent node to check if there are 0 or more child > nodes it is essentially trying to figure out the nature of state change > in ZK to determine whether to throttle or not. Subsequently it setsup > another watch in ZK to keep an eye on future changes. > When there are multiple bolts, there can be rapid creation/deletion of these > ZK nodes. Between the time the worker receives a callback and sets up the > next watch.. many changes may have undergone in ZK which will go unnoticed by > the spout. > The condition that the bolts are no longer congested may not get noticed as a > result of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1952) Keeping topology code for supervisor until topology got killed
[ https://issues.apache.org/jira/browse/STORM-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367670#comment-15367670 ] Robert Joseph Evans commented on STORM-1952: We do some of that already with the distributed cache, where we have an LRU cache for these objects. I would prefer for us to follow the same kind of policy that way we can keep the code around in case we need it, but we can also delete it if we need space and it has not been "used" in a long time (meaning no worker that needs it has run on this node for a while). Otherwise we risk running out of disk space. > Keeping topology code for supervisor until topology got killed > -- > > Key: STORM-1952 > URL: https://issues.apache.org/jira/browse/STORM-1952 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 1.0.0, 2.0.0, 1.0.1 >Reporter: Jungtaek Lim > > It's based on review comment from [~sriharsha]. > https://github.com/apache/storm/pull/1528/files#r69152524 > Please feel free to change reporter if you would like to. > In supervisor we're removing topology code when assignments for that > supervisor has gone. > But there's valid scenario to need to keep the topology code though > assignments for that supervisor is none, for example, rebalancing. > So it would be better for supervisor to keep topology code until topology has > been killed (and all topology workers assigned to that supervisor are also > killed). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1910) One topology can't use hdfs spout to read from two locations
[
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367664#comment-15367664
]
Robert Joseph Evans commented on STORM-1910:
I like the way that beam does this on a lot of their sources. They check by
default in the client, but offer a simple way to disable the check if needed.
This way we can have fail fast, but still provide compatibility with a wide
range of options.
> One topology can't use hdfs spout to read from two locations
>
>
> Key: STORM-1910
> URL: https://issues.apache.org/jira/browse/STORM-1910
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hdfs
>Affects Versions: 1.0.1
>Reporter: Raghav Kumar Gautam
>Assignee: Roshan Naik
> Fix For: 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
> conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
> if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
> LOG.error(Configs.SOURCE_DIR + " setting is required");
> throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
> }
> this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such
> errors at this time can only be detected at runtime by looking at the worker
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1879) Supervisor may not shut down workers cleanly
[
https://issues.apache.org/jira/browse/STORM-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344431#comment-15344431
]
Robert Joseph Evans commented on STORM-1879:
I have not seen either of these issue on our clusters.
I don't have a lot of time right now to try and reproduce it. I don't know how
much, if any critical information is going to be in the memory of nimbus/the
supervisor for your topology, but I don't suspect that it will be very much.
If any of you are OK with taking a heap dump of both nimbus and the supervisor
that is causing the issues when this happens it would probably be really
helpful.
The errors above happen occasionally because the cleanup of the files does not
always coincidence with shooting the worker so the worker could be
relauched/come up after the supervisor removed the config file it needs.
> Supervisor may not shut down workers cleanly
>
>
> Key: STORM-1879
> URL: https://issues.apache.org/jira/browse/STORM-1879
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.1
>Reporter: Stig Rohde Døssing
> Attachments: fix_missing_worker_pid.patch, nimbus-supervisor.zip,
> supervisor.log
>
>
> We've run into a strange issue with a zombie worker process. It looks like
> the worker pid file somehow got deleted without the worker process shutting
> down. This causes the supervisor to try repeatedly to kill the worker
> unsuccessfully, and means multiple workers may be assigned to the same port.
> The worker root folder sticks around because the worker is still heartbeating
> to it.
> It may or may not be related that we've seen Nimbus occasionally enter an
> infinite loop of printing logs similar to the below.
> {code}
> 2016-05-19 14:55:14.196 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyZendeskTicketTopology-5-1463647641-stormconf.ser
> 2016-05-19 14:55:14.210 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyZendeskTicketTopology-5-1463647641-stormcode.ser
> 2016-05-19 14:55:14.218 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyZendeskTicketTopology-5-1463647641-stormconf.ser
> 2016-05-19 14:55:14.256 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyZendeskTicketTopology-5-1463647641-stormcode.ser
> 2016-05-19 14:55:14.273 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyZendeskTicketTopology-5-1463647641-stormcode.ser
> 2016-05-19 14:55:14.316 o.a.s.b.BlobStoreUtils [ERROR] Could not update the
> blob with keyZendeskTicketTopology-5-1463647641-stormconf.ser
> {code}
> Which continues until Nimbus is rebooted. We also see repeating blocks
> similar to the logs below.
> {code}
> 2016-06-02 07:45:03.656 o.a.s.d.nimbus [INFO] Cleaning up
> ZendeskTicketTopology-127-1464780171
> 2016-06-02 07:45:04.132 o.a.s.d.nimbus [INFO]
> ExceptionKeyNotFoundException(msg:ZendeskTicketTopology-127-1464780171-stormjar.jar)
> 2016-06-02 07:45:04.144 o.a.s.d.nimbus [INFO]
> ExceptionKeyNotFoundException(msg:ZendeskTicketTopology-127-1464780171-stormconf.ser)
> 2016-06-02 07:45:04.155 o.a.s.d.nimbus [INFO]
> ExceptionKeyNotFoundException(msg:ZendeskTicketTopology-127-1464780171-stormcode.ser)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1890) Employ cache-busting method to ensure newly deployed UI forces browsers to refetch scripts, templates, and CSS
[
https://issues.apache.org/jira/browse/STORM-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320860#comment-15320860
]
Robert Joseph Evans commented on STORM-1890:
Oh yes for the templates. I forgot that we were fetching them using AJAX.
Thanks for pointing that out.
> Employ cache-busting method to ensure newly deployed UI forces browsers to
> refetch scripts, templates, and CSS
> --
>
> Key: STORM-1890
> URL: https://issues.apache.org/jira/browse/STORM-1890
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-ui
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Alessandro Bellina
>Priority: Minor
>
> Currently we don't employ cache busting techniques in the Storm UI while
> fetching script.js, CSS and templates. Ring is providing the Last-Modified
> header, but browsers implement a heuristic to when they deem a resource stale
> (https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.2.4). This
> means that as the Last-Modified for a resource is further away in the past,
> the longer the browsers are going to wait until they refetch. It looks like
> 10% padding is common, so if script.js was last modified 100 days ago, the
> browser will not fetch it until 10 days after the time it was cached.
> An easy approach is to add a url parameter to allow for cache busting
> whenever storm is packaged (mvn package). A more complicated method is
> versioning the files (we'd need to specify them in the pom.xml individually
> using the assembly plugin, unless we use some other plugin). The first method
> is (was?) considered less effective, since some CDNs/browsers can decide not
> to cache the query parameter.
> I'd like to go with the simpler method, unless there are strong opinions to
> changing file names (this means we need to specify files in the assembly
> pom.xml). Also, going this route we don't need any new plugins, and the
> assembly build can just be changed to export a variable. We would modify
> calls to include a value that changes on mvn package:
> {code}
>
> {code}
> instead of:
> {code}
>
> {code}
> Where $\{timestamp\} will be replaced at assembly time by maven. This would
> be the time when the assembly build started.
> The templates will also have the extra parameter. I think providing this to
> ajaxSetup will do the trick. For example:
> {code}
> $.ajaxSetup({ data: {"_ts" : "${timestamp}"}});
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1889) Datatables error message displayed when viewing UI
[
https://issues.apache.org/jira/browse/STORM-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320620#comment-15320620
]
Robert Joseph Evans commented on STORM-1889:
Sorry I think I put this in the wrong JIRA.
> Datatables error message displayed when viewing UI
> --
>
> Key: STORM-1889
> URL: https://issues.apache.org/jira/browse/STORM-1889
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-ui
>Affects Versions: 1.0.1
>Reporter: Simon Whittemore
>
> Updating to storm 1.0.1, running on Windows 7, I receive error messages from
> Datatables.
> This occurs on the Topology Summary as well as the Component Summary for a
> spout/bolt
> Example error: DataTables warning: table id=executor-stats-table - Requested
> unknown parameter '9' for row 0. For more information about this error,
> please see http://datatables.net/tn/4
> If I edit index.html to remove the type: num targets, the errors go away.
> For example.
> $.getJSON("/api/v1/topology/summary",function(response,status,jqXHR) {
> $.get("/templates/index-page-template.html", function(template) {
>
> topologySummary.append(Mustache.render($(template).filter("#topology-summary-template").html(),response));
> //name, owner, status, uptime, num workers, num executors, num
> tasks, replication count, assigned total mem, assigned total cpu, scheduler
> info
> dtAutoPage("#topology-summary-table", {
> columnDefs: [
> //{type: "num", targets: [4, 5, 6, 7, 8, 9]},
> {type: "num", targets: []},
> {type: "time-str", targets: [3]}
> ]
> });
> $('#topology-summary [data-toggle="tooltip"]').tooltip();
> });
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1759) Viewing logs from the Storm UI doesn't work in dockerized environment
[
https://issues.apache.org/jira/browse/STORM-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320621#comment-15320621
]
Robert Joseph Evans commented on STORM-1759:
I think we already have this ability.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/Config.java#L259-L268
By setting storm.local.hostname on each of the supervisors it should fix your
issue, because that is what will be reported to nimbus, and is what will be
used when building the URL for the logviewer.
> Viewing logs from the Storm UI doesn't work in dockerized environment
> -
>
> Key: STORM-1759
> URL: https://issues.apache.org/jira/browse/STORM-1759
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-ui
>Affects Versions: 0.10.0, 1.0.0, 0.9.6
>Reporter: Elisey Zanko
>
> I run the Storm using the following docker-compose.yml
> {code}
> version: '2'
> services:
> zookeeper:
> image: jplock/zookeeper:3.4.8
> restart: always
> nimbus:
> image: 31z4/storm:1.0.0
> command: nimbus -c storm.log.dir="/logs" -c
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - zookeeper
> restart: always
> ports:
> - 6627:6627
> volumes:
> - logs:/logs
> supervisor:
> image: 31z4/storm:1.0.0
> command: supervisor -c storm.log.dir="/logs" -c
> storm.zookeeper.servers="[\"zookeeper\"]" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> restart: always
> volumes:
> - logs:/logs
> logviewer:
> image: 31z4/storm:1.0.0
> command: logviewer -c storm.log.dir="/logs"
> restart: always
> ports:
> - 8000:8000
> volumes:
> - logs:/logs
> ui:
> image: 31z4/storm:1.0.0
> command: ui -c storm.log.dir="/logs" -c nimbus.host="nimbus"
> depends_on:
> - nimbus
> - logviewer
> restart: always
> ports:
> - 8080:8080
> volumes:
> - logs:/log
> volumes:
> logs: {}
> {code}
> And opening the logs from the Storm UI doesn't work because all links are
> pointing to different container ids as hosts.
> I guess adding an ability to explicitly specify the logviewer host in the
> storm.yaml would solve the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1889) Datatables error message displayed when viewing UI
[
https://issues.apache.org/jira/browse/STORM-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320616#comment-15320616
]
Robert Joseph Evans commented on STORM-1889:
I think we already have this ability.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/Config.java#L259-L268
By setting storm.local.hostname on each of the supervisors it should fix your
issue, because that is what will be reported to nimbus, and is what will be
used when building the URL for the logviewer.
> Datatables error message displayed when viewing UI
> --
>
> Key: STORM-1889
> URL: https://issues.apache.org/jira/browse/STORM-1889
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-ui
>Affects Versions: 1.0.1
>Reporter: Simon Whittemore
>
> Updating to storm 1.0.1, running on Windows 7, I receive error messages from
> Datatables.
> This occurs on the Topology Summary as well as the Component Summary for a
> spout/bolt
> Example error: DataTables warning: table id=executor-stats-table - Requested
> unknown parameter '9' for row 0. For more information about this error,
> please see http://datatables.net/tn/4
> If I edit index.html to remove the type: num targets, the errors go away.
> For example.
> $.getJSON("/api/v1/topology/summary",function(response,status,jqXHR) {
> $.get("/templates/index-page-template.html", function(template) {
>
> topologySummary.append(Mustache.render($(template).filter("#topology-summary-template").html(),response));
> //name, owner, status, uptime, num workers, num executors, num
> tasks, replication count, assigned total mem, assigned total cpu, scheduler
> info
> dtAutoPage("#topology-summary-table", {
> columnDefs: [
> //{type: "num", targets: [4, 5, 6, 7, 8, 9]},
> {type: "num", targets: []},
> {type: "time-str", targets: [3]}
> ]
> });
> $('#topology-summary [data-toggle="tooltip"]').tooltip();
> });
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1890) Employ cache-busting method to ensure newly deployed UI forces browsers to refetch scripts, templates, and CSS
[
https://issues.apache.org/jira/browse/STORM-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320607#comment-15320607
]
Robert Joseph Evans commented on STORM-1890:
That sounds great. I don't think we need that for the ajaxSetup. We have not
seen issues where the REST calls are cached, I think jquery disables caching
automatically for all of the ajax calls, but I am no expert here.
> Employ cache-busting method to ensure newly deployed UI forces browsers to
> refetch scripts, templates, and CSS
> --
>
> Key: STORM-1890
> URL: https://issues.apache.org/jira/browse/STORM-1890
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-ui
>Affects Versions: 2.0.0, 1.1.0
>Reporter: Alessandro Bellina
>Priority: Minor
>
> Currently we don't employ cache busting techniques in the Storm UI while
> fetching script.js, CSS and templates. Ring is providing the Last-Modified
> header, but browsers implement a heuristic to when they deem a resource stale
> (https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.2.4). This
> means that as the Last-Modified for a resource is further away in the past,
> the longer the browsers are going to wait until they refetch. It looks like
> 10% padding is common, so if script.js was last modified 100 days ago, the
> browser will not fetch it until 10 days after the time it was cached.
> An easy approach is to add a url parameter to allow for cache busting
> whenever storm is packaged (mvn package). A more complicated method is
> versioning the files (we'd need to specify them in the pom.xml individually
> using the assembly plugin, unless we use some other plugin). The first method
> is (was?) considered less effective, since some CDNs/browsers can decide not
> to cache the query parameter.
> I'd like to go with the simpler method, unless there are strong opinions to
> changing file names (this means we need to specify files in the assembly
> pom.xml). Also, going this route we don't need any new plugins, and the
> assembly build can just be changed to export a variable. We would modify
> calls to include a value that changes on mvn package:
> {code}
>
> {code}
> instead of:
> {code}
>
> {code}
> Where $\{timestamp\} will be replaced at assembly time by maven. This would
> be the time when the assembly build started.
> The templates will also have the extra parameter. I think providing this to
> ajaxSetup will do the trick. For example:
> {code}
> $.ajaxSetup({ data: {"_ts" : "${timestamp}"}});
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-1490) Update storm-merge.py to be full featured
[ https://issues.apache.org/jira/browse/STORM-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-1490: --- Assignee: (was: Robert Joseph Evans) > Update storm-merge.py to be full featured > - > > Key: STORM-1490 > URL: https://issues.apache.org/jira/browse/STORM-1490 > Project: Apache Storm > Issue Type: Improvement > Components: build >Reporter: Robert Joseph Evans > > storm-merge.py should optionally do more then just print out the merge > command. > At a minimum it should >1. update the branch you are trying to merge to >2. create the branch integration branch (based off of the first branch) >3. run the merge >4. update the changelog (unless we just want this to go away) >5. run some sanity tests (minimum build, ideally some tests and RAT too) >6. merge the integration back to the main branch >7. push the result back to apache > It would also really be nice if it could offer the ability to cherry-pick a > pull request made on another branch. Updating things as needed > If we want the changelog to go away it would not be too hard to combine a > JIRA query with a git log to regenerate that too. But that might be a > separate tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1772) Create topologies for measuring performance
[ https://issues.apache.org/jira/browse/STORM-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289081#comment-15289081 ] Robert Joseph Evans commented on STORM-1772: Sorry, yes I want to move it forward. Life happens, but I will try to make time soon to pull in the current pull requests. > Create topologies for measuring performance > --- > > Key: STORM-1772 > URL: https://issues.apache.org/jira/browse/STORM-1772 > Project: Apache Storm > Issue Type: Bug >Reporter: Roshan Naik >Assignee: Roshan Naik > > Would be very useful to have some simple reference topologies included with > Storm that can be used to measure performance both by devs during development > (to start with) and perhaps also on a real storm cluster (subsequently). > To start with, the goal is to put the focus on the performance > characteristics of individual building blocks such as specifics bolts, > spouts, grouping options, queues, etc. So, initially biased towards > micro-benchmarking but subsequently we could add higher level ones too. > Although there is a storm benchmarking tool (originally written by Intel?) > that can be used, and i have personally used it, its better for this to be > integrated into Storm proper and also maintained by devs as storm evolves. > On a side note, in some instances I have noticed (to my surprise) that the > perf numbers change when the topologies written for Intel benchmark when > rewritten without the required wrappers so that they runs directly under > Storm. > Have a few topologies in mind for measuring each of these: > # *Queuing and Spout Emit Performance:* A topology with a Generator Spout but > no bolts. > # *Queuing & Grouping performance*: Generator Spout -> A grouping method -> > DevNull Bolt > # *Hdfs Bolt:*Generator Spout -> Hdfs Bolt > # *Hdfs Spout:* Hdfs Spout -> DevNull Botl > # *Kafka Spout:* Kafka Spout -> DevNull Bolt > # *Simple Data Movement*: Kafka Spout -> Hdfs Bolt > Shall add these for Storm core first. Then we can have the same for Trident > also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287241#comment-15287241 ] Robert Joseph Evans commented on STORM-1757: OK so that was not clear from the javadocs in state. I should have read through more of the documentation that you sent out. Thanks for putting up with some of my crazy questions. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286933#comment-15286933 ] Robert Joseph Evans commented on STORM-1757: [~arunmahadevan], BEAM does exactly once, by doing at least once, but making sure that the checkpoints happen in an idempotent and coordinated way. The issue is not at least once vs exactly once, but is around failure cases and consistency. From what I have seen with the state API + windowing offers some coordination and restore, reprocess, re-checkpoint should be idempotent, but it can only role back to the last successful checkpoint. So if I ask all bolts to checkpoint and most succeed but one does not, I can not longer restore everyone to a consistent place, so I can start replaying again. This is what I am concerned about. I would like to be able to fully support the BEAM model, and then if we can offer better performance by reducing guarantees I am OK with that. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1599) Don't mark dependencies as provided unless they are in lib
[
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated STORM-1599:
---
Description:
When we mark a dependency as provided it indicates the shade and assembly
plugins to not include this particular dependency in the uber topology jar
because it will be {{provided}} on the class path by the system.
We have been doing this for all of our kafka dependencies incorrectly,
storm-cassandra does this for cassandra-driver-core, and storm-starter is doing
it for storm-clojure as well.
This means that storm-starter does not have any version of kafka or
storm-clojure packaged it the resulting jar and any example that uses kafka,
TridentKafkaWordCount, will fail with missing class errors.
storm-starter/pom.xml has should change its dependency on storm-kafka to be
compile, and it should delete dependencies on kafka and kafka-clients as those
should come from storm-kafka as transitive dependencies.
the main pom.xml should not have kafka-clients marked as provided in the
dependency management section.
storm-kafka should remove its provided tag on kafka, and flux examples +
storm-sql-kafka should remove dependencies on kafka and kafka-clients, and
storm-kafka should not me marked as provided.
the flux and sql code I am not as familiar with, but looking at them, and
running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like
was:
When we mark a dependency as provided it indicates the shade and assembly
plugins to not include this particular dependency in the uber topology jar
because it will be {provided} on the class path by the system.
We have been doing this for all of our kafka dependencies incorrectly. This
means that storm-starter does not have any version of kafka packaged it the
resulting jar and any example that uses kafka, TridentKafkaWordCount, will fail
with missing class errors.
storm-starter/pom.xml has should change its dependency on storm-kafka to be
compile, and it should delete dependencies on kafka and kafka-clients as those
should come from storm-kafka as transitive dependencies.
the main pom.xml should not have kafka-clients marked as provided in the
dependency management section.
storm-kafka should remove its provided tag on kafka, and flux examples +
storm-sql-kafka should remove dependencies on kafka and kafka-clients, and
storm-kafka should not me marked as provided.
the flux and sql code I am not as familiar with, but looking at them, and
running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like
> Don't mark dependencies as provided unless they are in lib
> --
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
> Issue Type: Bug
> Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly
> plugins to not include this particular dependency in the uber topology jar
> because it will be {{provided}} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly,
> storm-cassandra does this for cassandra-driver-core, and storm-starter is
> doing it for storm-clojure as well.
> This means that storm-starter does not have any version of kafka or
> storm-clojure packaged it the resulting jar and any example that uses kafka,
> TridentKafkaWordCount, will fail with missing class errors.
> storm-starter/pom.xml has should change its dependency on storm-kafka to be
> compile, and it should delete dependencies on kafka and kafka-clients as
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples +
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and
> storm-kafka should not me marked as provided.
> the flux and sql code I am not as familiar with, but looking at them, and
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1599) Kafka dependencies all marked as provided (so storm-starter does not run)
[
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286905#comment-15286905
]
Robert Joseph Evans commented on STORM-1599:
I think that marking them as provided is the outlier. The majority of storm-*
don't mark their dependencies as provided. In general though we should work
towards doing a much better job of managing the system classpath, dependencies,
and making it consistent everywhere. Hopefully we can do that as part of this
JIRA.
Right now the only things that storm "provides" on the system classpath are
storm-core and it's non-shaded dependencies, and storm-rename-hack. Nothing
else is truly provided by storm.
I ran {{find . -iname pom.xml | xargs grep -B 3 provided | grep artifactId |
egrep -v 'storm-core|storm-rename-hack'}} to find everywhere that we list
something as provided that really isn't
{code}
./examples/storm-starter/pom.xml-storm-clojure
./examples/storm-starter/pom.xml- storm-kafka
./examples/storm-starter/pom.xml-
${kafka.artifact.id}
./external/sql/storm-sql-core/pom.xml-
storm-sql-runtime
./external/sql/storm-sql-kafka/pom.xml-
storm-sql-runtime
./external/sql/storm-sql-kafka/pom.xml-
storm-kafka
./external/sql/storm-sql-kafka/pom.xml-
${kafka.artifact.id}
./external/storm-cassandra/pom.xml-
cassandra-driver-core
./external/storm-kafka/pom.xml-
${kafka.artifact.id}
./storm-buildtools/maven-shade-clojure-transformer/pom.xml-
maven-shade-plugin
./storm-buildtools/storm-maven-plugins/pom.xml-
maven-plugin-annotations
{code}
storm-clojure is new in 2.0 and should not be provided. It is not in lib.
storm-kafka and kafka.artifact.id is what this jira is about. storm-sql-* had
some issues where it was on the classpath even though it should not have been,
so they need to be updated accordingly too.
Looks like storm-cassandra should have the cassandra-driver-core fixed as well.
Then all that is left is the build-tools, which are not part of a topology jar
and can be ignored.
As such I will update the title of this JIRA to indicate that we should fix
storm-cassandra and strom-starter's dependency on storm-clojure.
> Kafka dependencies all marked as provided (so storm-starter does not run)
> -
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
> Issue Type: Bug
> Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly
> plugins to not include this particular dependency in the uber topology jar
> because it will be {provided} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly. This
> means that storm-starter does not have any version of kafka packaged it the
> resulting jar and any example that uses kafka, TridentKafkaWordCount, will
> fail with missing class errors.
> storm-starter/pom.xml has should change its dependency on storm-kafka to be
> compile, and it should delete dependencies on kafka and kafka-clients as
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples +
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and
> storm-kafka should not me marked as provided.
> the flux and sql code I am not as familiar with, but looking at them, and
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-1599) Don't mark dependencies as provided unless they are in lib
[
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated STORM-1599:
---
Summary: Don't mark dependencies as provided unless they are in lib (was:
Kafka dependencies all marked as provided (so storm-starter does not run))
> Don't mark dependencies as provided unless they are in lib
> --
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
> Issue Type: Bug
> Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly
> plugins to not include this particular dependency in the uber topology jar
> because it will be {provided} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly. This
> means that storm-starter does not have any version of kafka packaged it the
> resulting jar and any example that uses kafka, TridentKafkaWordCount, will
> fail with missing class errors.
> storm-starter/pom.xml has should change its dependency on storm-kafka to be
> compile, and it should delete dependencies on kafka and kafka-clients as
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples +
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and
> storm-kafka should not me marked as provided.
> the flux and sql code I am not as familiar with, but looking at them, and
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286713#comment-15286713 ] Robert Joseph Evans commented on STORM-1757: [~satish.duggana], I'm not totally sure if this is the right place to ask this or not. If you want me to move it to STORM-1763 I ma happy to. How do we plan on doing a coordinated restore using the current [State|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/state/State.java] API? I supposed we can extend it so rollback optionally supports a commit id, and there is an API to release commits older than a give commit id that are not needed any more. Should I file a separate JIRA for something like that? Unless we want to start out with the simpler one micro batch at a time processing. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286704#comment-15286704 ] Robert Joseph Evans commented on STORM-1757: [~satish.duggana], That all sounds great. So what is really left is coordination between the sources and a translation layer, followed by lots and lots of optimizations/testing. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284881#comment-15284881 ] Robert Joseph Evans commented on STORM-1757: [~arunmahadevan], Great to hear that at least some of it can be reused. Will have to dig into it more to see exactly how much we can reuse. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284871#comment-15284871 ] Robert Joseph Evans commented on STORM-1757: I agree that going to a new API and deprecating other APIs is something that we need to do thoughtfully and is not going to happen overnight. Perhaps it warrants it's own JIRA for discussion/exploration. Having multiple APIs is a pain from both a maintenance standpoint and from a user standpoint in deciding what to do. You mean I have to rewrite things if I don't need exactly once any more and I want better performance? It would be nice to be able to support exactly once processing in a unified API with at least once and at most once. Preferably something that is backwards compatible with the existing storm core API. But I really haven't spent much time thinking through how that API would work/look. For the most part I would expect it could look like the check-pointing code we have now, but some de-duping happening in there as well. Either way I think this needs to be on a separate JIRA so we can discuss a long term plan for storm. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284755#comment-15284755 ] Robert Joseph Evans commented on STORM-1757: Sorry about the long posts, but I think I am done and have most of what I have been thinking about this week out of my head. Any feedback/suggestions would be greatly appreciated. (I have not thought through merging windows and how the check-pointing would work just yet, but I think it then becomes metadata around the panes and deciding which set of panes to read instead) > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284752#comment-15284752 ] Robert Joseph Evans commented on STORM-1757: To make distributed check pointing work we need more coordination between the Sources and the bolts than ack/fail really offers. For this I would propose that we have each Unbounded Source run as a spout. All of the spouts would need to coordinate with each other so that the barriers can be emitted close to one another, reducing the amount of data that would need to be buffered. We would also need coordination for restoring checkpoints. For this I would propose that we use zookeeper for this. Simply because it is already available and we would not need to add anything new to base storm to support it. All of the spouts would elect a leader through zookeeper. The leader would then trigger all of the spouts to emit a barrier and checkpoint the spout metadata. Because we are going to potentially have multiple checkpoints outstanding at any point in time we will need to label all of the checkpoints. I would label them with two numbers. The first would be the bundle/batch number, the second would be the replay, or generation number. The bundle number would increment for each barrier emitted, but would role back on failure. The generation number would increment for any failure. This would allow downstream bolts to be able to restore a checkpoint just by seeing the bundle id. Spouts would have acking to know if a downstream tuple failed. If an item fails the spout would inform the leader through zookeeper of the bad batch. The leader would then inform all of the other spouts to restore and start again. Each spout would also have to inform the leader periodically when a batch is fully acked. Once all of the spouts inform the leader that all of the tuples are emitted, then the leader can inform the spouts that they can delete the old checkpoints. They should also inform the downstream bolts as part of the barrier so they can clean up their old checkpoints too. We can work out the exact details of how this will work later on. For the actual check pointing in the bolts only the GroupByKey transform would need to do anything, and for simplicity it would checkpoint each pane and key separately, so that the checkpoints are incremental, and so that we can support very large windows without too much difficulty. In general all of this seems totally doable, and actually not that difficult. My biggest concern by far is around efficiency in the check pointing, especially for large windows. The check pointing is something that we need to do, and in the common case should be thrown away. So we want to be sure that we optimize for throwing the data away. We can easily write something that can be backed by HBase or most any nosql store. But that is going to add a lot of iops and network load that I am not too thrilled about. But perhaps it does not really matter for an initial deployment. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284682#comment-15284682 ] Robert Joseph Evans commented on STORM-1757: BEAM logically has a few base transforms, that we need to support. We can extend this in the future to support more if we need/want to. * *GroupByKey* - Groups by key and window * CreatePCollectionView * Read.Bounded - only for batch (which we probably can support, just not right now) * *Read.Unbounded* * *ParDo.Bound* * ParDo.BoundMulti * *FlattenPCollectionList* - a merge of PCollections (Streams) * *Window.Bound* The *Bold* ones are what I think we should concentrate on first, and then we can add in support for side inputs and outputs in a second phase, and finally add in support for bounded "batch" processing if we see a need to. Like I stated before fault tolerance in an UnboundedSource is built around a checkpoint restore model. While reading data we can ask the source for metadata that we can then use to restore the source back to a given point in time. Windowing in BEAM requires check pointing and is based around the concept of a pane (which is a part of a window) and triggers. The GroupByKey transform is special in that it not only groups by a key, but also groups by a window. To do this windowed data will not be emitted until a trigger fires to start the processing. And even when the trigger fires there is a choice to discard the panes that have already been processed or to retain them until the trigger indicates that it will never fire again. Triggers usually are based off of a watermark that comes from the source. All ParDos also have a concept around "bundles" that are batches of tuples being processed. The DoFns can be informed of the start and end of a bundle, and in non-optimized cases will be recreated for each bundle. When we combine all of this together, we need a way to checkpoint/restore all of the state in a consistent way across the entire topology, preferably coordinated with the bundles as well. We could do this one of two ways. The simplest approach would be to have a trident like coordinator that # Tells bolts to prepare for a new batch # wait for ack # Tells spouts to emit a new batch # wait for ack # Tells everyone to checkpoint # wait for ack On success repeat. On failure: # Tell everyone to roll back to last successful checkpoint # Wait for ack # Start again This would be a simple way to get things started, but it would be very slow because we cannot have more then one batch of tuples outstanding at any point in time, and even if we can combine a few steps together we will need at least two round trips through the topology to finish a batch. If we want to try and go faster it is going to require more of an exchange of metadata, and check-pointing similar to how flink or apex do it. https://ci.apache.org/projects/flink/flink-docs-master/internals/stream_checkpointing.html Perhaps this is something that we should think about once we have a prototype working with the first implementation. More on the distributed check pointing coming shortly... > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284636#comment-15284636 ] Robert Joseph Evans commented on STORM-1757: I have gone through the BEAM API and I think I understand it fairly well now. I have a plan on how we could implement this, but honestly it is a different enough model from trident, that I don't think the two will be able to reuse code, like I stated before. Additionally I have not dug into our windowing code to know how it might work with BEAM, so any feedback on how if it can be reused would be appreciated. Short term I would propose that we go ahead with a BEAM implementation that runs on storm, not trident. Longer term I believe we should move towards a three tiered API approach. This is what most compiler frameworks use, and I think makes a lot of since in this use case too. The lowest level is like assembly language. For storm this would be bolts, spouts, groupings, and possibly some other things around state, check-pointing and coordination. This is what the worker and most of the scheduler sees and executes. There would also be an intermediate representation that describes the logical operations being performed. These would be fairly simple logical operations link map, groupbykey, windowing, etc. An optimizer would take this as input and transform into an optimized solution. Eventually this would be a cost based optimizer that would also take metrics from the running code + hints to understand where data is actually flowing, how skewed is the data, etc to improve the plan over time. A "compiler" would then translate the optimized intermediate representation into the assembled topology. The scheduler could then place the code physically on the cluster it is running on. High level languages like SQL, BEAM, Trident, etc. would translate their operations into the intermediate representation and submit it to nimbus for execution. This would let us keep the regular storm API, but also be able to maintaqin a few "standard" high level languages and support multiple domain specific languages. BEAM design coming shortly. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282938#comment-15282938 ] Robert Joseph Evans commented on STORM-1757: Yes it will be a much larger effort than just doing a BEAM runner. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281994#comment-15281994 ] Robert Joseph Evans commented on STORM-1757: I agree that having more than one official Storm API is a support/maintenance issue, and worse it is confusing for customers. But I also see storm as a base platform that others can build APIs/Languages on top of. If someone wants an X compatible API on top of storm we should encourage them, and possibly add in features to core storm to support them, but we don't necessarily pull them into our repo. I see beam and SQL in this context. If you don't like trident, or think we can replace it with some changes to the normal storm API then lets think about what the correct API should be and implement it. If beam really is the correct API and we don't think we can do better, then storm should look at becoming the reference implementation for open source beam, and we should deprecate both Trident and the core storm API. But I personally don't think it is. From what I have seen the beam API is designed very much around ETL. It does that very well. It can handle complex windowing for you and make the job of most of my customers rather simple. This is why I am exploring beam compatibility. But in the case of micro services like what kafka streams appears to be targeting, or in the case IOT, is beam the right API for that? It may grow up to be eventually, but right now it does not appear to be at all. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281907#comment-15281907 ] Robert Joseph Evans commented on STORM-1757: [~sriharsha], Sorry I misread you original comment. You don't want to merge the two you want BEAM to replace trident. That seems possible, but it is definitely not a 1 to 1 translation, and Honestly I find the BEAM API much more difficult to get started on than I do spark, flink, or even the tirdent APIs. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281905#comment-15281905 ] Robert Joseph Evans commented on STORM-1757: I would love to have something like that happen, but I am not sure if the two are even compatible at a fundamental level. I think the Source/Spout APIs are a good example of this. https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/trident/spout/ITridentSpout.java Provides two distinct things a BatchCoordinator and an Emitter. https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java Really just provides a single thing, an UnboundedReader. Both have a bit of metadata that is intended to act as a checkpoint of where they are currently at in processing (X in Trident and CheckpointMark in BEAM), but what they represent is fundamentally different. In Trident X represents a complete batch of data for processing. A start point through to an end point. In BEAM a CheckpointMark is just a place in the processing. It is a spot where a checkpoint can be restored to. There is no end to it, there is no batch. I can turn it into a batch if I stop reading after some amount of time, or after a given amount of data, but then I am just at a new point. State/Sinks are another. In Trident a State https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/trident/state/State.java is a specific thing that can have data written to it and in some cases queried from it. But ultimately it really just comes down to two APIs begineTransaction and commit transaction. Transactions to state are guaranteed to be committed in order. Beam has not such concept the closest logically to it is a Sink, but a Sink is just for batch, in streaming you just use an idempotent ParDo. But in reality every ParDo offers similar lifecycle operations to a State. https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java A DoFn runs inside a ParDo, and has APIs for startBundle and finishBundle. A bundle is an arbitrary batch of data being processed, it seems that in streaming it is intended to be used around check-pointing, but there is no id like with state, so there actually is no guarantee of the order of processing. In fact it is quite the opposite they want to be sure that ParDos can even run in parallel on different machines. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281795#comment-15281795 ] Robert Joseph Evans commented on STORM-1757: I have a customer that is potentially interested in this too, but I need to come up with a plan/LOE to give them some advice. For the most part it seems like we should be able to build something fairly quickly, but like you said things don't seem too stable over on the BEAM side. I am going to probe a bit more on the mailing lists to try and understand some of the details in Windowing/Triggering, and side data. There are a few primitive operations that we need to implement a translation/runner for. GroupByKey - Groups by key and window CreatePCollectionView - for side input, not sure how/if this works with streaming Read.Bounded - only for batch (which we probably can support) Read.Unbounded ParDo.Bound ParDo.BoundMulti FlattenPCollectionList Window.Bound I am still working things out, but the main difference between what we have done so far and what we have right now with trident windowing/checkpointing is around the triggers and the watermarks. I don't know the internals of the windowing/checkpointing work in storm well enough to really say that if we had a trident implementation it would work, but I don't think so. I still have a bit more I need to understand before I can come up with a final design. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1772) Create topologies for measuring performance
[ https://issues.apache.org/jira/browse/STORM-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280160#comment-15280160 ] Robert Joseph Evans commented on STORM-1772: I would love to see more performance tests/topologies in storm. I wrote one https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/ThroughputVsLatency.java but it is just a word count topology, and it requires acking to be enabled for it to work properly. We also did a storm benchmark as part of https://github.com/yahoo/streaming-benchmarks. And then there is the https://github.com/yahoo/storm-perf-test which is outdated, and too many people use it to test the general performance of storm instead of using it as stress test on the messaging layer, which is the only thing it is really good at. I think ThroughputVsLatency is the correct model that we want to follow for these types of tests. Trying to capture the processing latency at a given throughput. It can easily be adapted to other topologies, and with a small amount of work should be able to support an OK measure of latency without the need for acking, assuming that the clocks among the different nodes are close to being in sync with one another. The intel benchmarks are great in their diversity of workloads, but I would also like to see something that is more real world too. > Create topologies for measuring performance > --- > > Key: STORM-1772 > URL: https://issues.apache.org/jira/browse/STORM-1772 > Project: Apache Storm > Issue Type: Bug >Reporter: Roshan Naik >Assignee: Roshan Naik > > Would be very useful to have some simple reference topologies included with > Storm that can be used to measure performance both by devs during development > (to start with) and perhaps also on a real storm cluster (subsequently). > To start with, the goal is to put the focus on the performance > characteristics of individual building blocks such as specifics bolts, > spouts, grouping options, queues, etc. So, initially biased towards > micro-benchmarking but subsequently we could add higher level ones too. > Although there is a storm benchmarking tool (originally written by Intel?) > that can be used, and i have personally used it, its better for this to be > integrated into Storm proper and also maintained by devs as storm evolves. > On a side note, in some instances I have noticed (to my surprise) that the > perf numbers change when the topologies written for Intel benchmark when > rewritten without the required wrappers so that they runs directly under > Storm. > Have a few topologies in mind for measuring each of these: > # *Queuing and Spout Emit Performance:* A topology with a Generator Spout but > no bolts. > # *Queuing & Grouping performance*: Generator Spout -> A grouping method -> > DevNull Bolt > # *Hdfs Bolt:*Generator Spout -> Hdfs Bolt > # *Hdfs Spout:* Hdfs Spout -> DevNull Botl > # *Kafka Spout:* Kafka Spout -> DevNull Bolt > # *Simple Data Movement*: Kafka Spout -> Hdfs Bolt > Shall add these for Storm core first. Then we can have the same for Trident > also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm
[ https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276656#comment-15276656 ] Robert Joseph Evans commented on STORM-1757: [~ptgoetz] and others, I am fine no matter where we put it for initial development. Putting on my Yahoo hat for a bit. We are interested in helping out on this too, but we have a lot of other work/priorities that we are trying to plan out, at least at a high level. Has any work gone into designing how we would plan to support this? Is it a layer on top of trident with the windowing support added in? I have not had the time to dig into the beam API that fully so I am not really sure what other gotchas might crop up. I really would like to get a handle on this so I can plan accordingly. > Apache Beam Runner for Storm > > > Key: STORM-1757 > URL: https://issues.apache.org/jira/browse/STORM-1757 > Project: Apache Storm > Issue Type: Brainstorming >Reporter: P. Taylor Goetz >Priority: Minor > > This is a call for interested parties to collaborate on an Apache Beam [1] > runner for Storm, and express their thoughts and opinions. > Given the addition of the Windowing API to Apache Storm, we should be able to > map naturally to the Beam API. If not, it may be indicative of shortcomings > of the Storm API that should be addressed. > [1] http://beam.incubator.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-841) Thread-safeness of OutputCollector has documented contrary to two official doc.
[
https://issues.apache.org/jira/browse/STORM-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262275#comment-15262275
]
Robert Joseph Evans commented on STORM-841:
---
When is it not thread safe any more?
I thought the only thing that made it not thread safe was the shuffle grouping,
which was fixed when the load aware shuffle grouping code went in.
https://github.com/apache/storm/blob/v1.0.1/storm-core/src/jvm/org/apache/storm/grouping/ShuffleGrouping.java#L61
that went into the 1.0 release STORM-162.
For 0.10 and 9.6 it is not thread safe but from 1.0 on-wards I think it is.
> Thread-safeness of OutputCollector has documented contrary to two official
> doc.
> ---
>
> Key: STORM-841
> URL: https://issues.apache.org/jira/browse/STORM-841
> Project: Apache Storm
> Issue Type: Documentation
> Components: documentation
>Affects Versions: 0.10.0, 1.0.0, 0.9.6, 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> There're some issues with documentation.
> http://storm.apache.org/documentation/Concepts.html says
> {quote}
> Its perfectly fine to launch new threads in bolts that do processing
> asynchronously. OutputCollector is thread-safe and can be called at any time.
> {quote}
> and http://storm.apache.org/documentation/Troubleshooting.html says
> {quote}
> This is caused by having multiple threads issue methods on the
> OutputCollector. All emits, acks, and fails must happen on the same thread.
> One subtle way this can happen is if you make a IBasicBolt that emits on a
> separate thread. IBasicBolt's automatically ack after execute is called, so
> this would cause multiple threads to use the OutputCollector leading to this
> exception. When using a basic bolt, all emits must happen in the same thread
> that runs execute.
> {quote}
> It is a contradiction, and at least for now OutputCollector is not
> thread-safe.
> https://www.mail-archive.com/dev@storm.incubator.apache.org/msg00939.html
> Since newbie of Storm users may think Concepts page as "should read and keep
> it mind", it is some kind of critical that that such important documentation
> page has wrong content.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1740) Storm on mesos and yarn
[ https://issues.apache.org/jira/browse/STORM-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262161#comment-15262161 ] Robert Joseph Evans commented on STORM-1740: That is fine with me in some use cases, but we don't want to lose the ability to run standalone too. > Storm on mesos and yarn > --- > > Key: STORM-1740 > URL: https://issues.apache.org/jira/browse/STORM-1740 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Renjie Liu >Priority: Minor > > Storm should delegate resource management to mesos or yarn rather than use it > own schemanism. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-143) Launching a process throws away standard out; can hang
[
https://issues.apache.org/jira/browse/STORM-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15260170#comment-15260170
]
Robert Joseph Evans commented on STORM-143:
---
If you know the file that was changed, or some code that was changed as a part
of it, both blame and history can be big helps in getting the traceability. We
don't squash commits though so it can sometimes not give you what you want.
But in this case it works out OK, but you may need to expand the commit comment
to find the JIRA number
https://github.com/apache/storm/commits/0.10.x-branch/storm-core/src/jvm/backtype/storm/LogWriter.java
STORM-833
and the pull request for it shows LogWriter being added.
https://github.com/apache/storm/pull/562/files
> Launching a process throws away standard out; can hang
> --
>
> Key: STORM-143
> URL: https://issues.apache.org/jira/browse/STORM-143
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Reporter: James Xu
>Priority: Minor
>
> https://github.com/nathanmarz/storm/issues/489
> https://github.com/nathanmarz/storm/blob/master/src/clj/backtype/storm/util.clj#L349
> When we launch a process, standard out is written to a system buffer and does
> not appear to be read. Also, nothing is redirected to standard in. This can
> have the following effects:
> A worker can hang when initializing (e.g. UnsatisfiedLinkError looking for
> jzmq), and it will be unable to communicate the error as standard out is
> being swallowed.
> A process that writes too much to standard out will block if the buffer fills
> A process that tries to read form standard in for any reason will block.
> Perhaps we can redirect standard out to an .out file, and redirect /dev/null
> to the standard in stream of the process?
> --
> nathanmarz: Storm redirects stdout to the logging system. It's worked fine
> for us in our topologies.
> --
> d2r: We see in worker.clj, in mk-worker, where there is a call to
> redirect-stdio-to-slf4j!. This would not seem to help in cases such as we are
> seeing when there is a problem launching the worker itself.
> (defn -main [storm-id assignment-id port-str worker-id]
> (let [conf1 (read-storm-config)
> login_conf_file (System/getProperty "java.security.auth.login.config")
> conf (if login_conf_file (merge conf1
> {"java.security.auth.login.config" login_conf_file}) conf1)]
> (validate-distributed-mode! conf)
> (mk-worker conf nil (java.net.URLDecoder/decode storm-id) assignment-id
> (Integer/parseInt port-str) worker-id)))
> If anything were to go wrong (CLASSPATH, jvm opts, misconfiguration...)
> before -main or before mk-worker, then any output would be lost. The symptom
> we saw was that the topology sat around apparently doing nothing, yet there
> was no log indicating that the workers were failing to start.
> Is there other redirection to logs that I'm missing?
> --
> xiaokang: we use bash to launch worker process and redirect its stdout to
> woker-port.out file. it heleped us find the zeromq jni problem that cause the
> jvm crash without any log.
> --
> nathanmarz: @d2r Yea, that's all I was referring to. If we redirect stdout,
> will the code that redirects stdout to the logging system still take effect?
> This is important because we can control the size of the logfiles (via the
> logback config) but not the size of the redirected stdout file.
> --
> d2r: My hunch is that it will work as it does now, except that any messages
> that are getting thrown away before that point would go to a file instead. I
> can play with it and find out. We wouldn't want to change the redirection,
> just restore visibility to any output that might occur prior to the
> redirection. There should be some safety valve to control the size of any new
> .out in case something goes berserk.
> @xiaokang I see how that would work. We also need to make sure redirection
> continues to work as it currently does for the above reason.
> --
> xiaokang: @d2r @nathanmarz In out cluster, storm's stdout redirection still
> works for any System.out output while JNI errors goes to worker-port.out
> file. I think it will be nice to use the same worker-port.log file for bash
> stdout redirection since logback can control log file size. But it is a
> little bit ugly to use bash to launch worker java process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (STORM-1710) java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
[
https://issues.apache.org/jira/browse/STORM-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-1710.
Resolution: Invalid
Assignee: Robert Joseph Evans
Fix Version/s: (was: 1.0.0)
> java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
> -
>
> Key: STORM-1710
> URL: https://issues.apache.org/jira/browse/STORM-1710
> Project: Apache Storm
> Issue Type: Question
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Linux 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41
> EST 2013 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Guangxin Zhang
>Assignee: Robert Joseph Evans
>
> 2016-04-14 09:07:22.226 o.a.s.d.supervisor [ERROR] Error on initialization of
> server mk-supervisor
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> backtype.storm.generated.LSSupervisorId
> at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83)
> at org.apache.storm.utils.LocalState.get(LocalState.java:130)
> at
> org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61)
> at
> org.apache.storm.daemon.supervisor$standalone_supervisor$reify__9646.prepare(supervisor.clj:1228)
> at
> org.apache.storm.daemon.supervisor$fn__9503$exec_fn__1826__auto9504.invoke(supervisor.clj:781)
> at clojure.lang.AFn.applyToHelper(AFn.java:160)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:630)
> at
> org.apache.storm.daemon.supervisor$fn__9503$mk_supervisor__9548.doInvoke(supervisor.clj:779)
> at clojure.lang.RestFn.invoke(RestFn.java:436)
> at
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1216)
> at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1249)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.supervisor.main(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
> backtype.storm.generated.LSSupervisorId
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:186)
> at org.apache.storm.utils.LocalState.deserialize(LocalState.java:78)
> ... 14 more
> 2016-04-14 09:07:22.239 o.a.s.util [ERROR] Halting process: ("Error on
> initialization")
> java.lang.RuntimeException: ("Error on initialization")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at
> org.apache.storm.daemon.supervisor$fn__9503$mk_supervisor__9548.doInvoke(supervisor.clj:779)
> at clojure.lang.RestFn.invoke(RestFn.java:436)
> at
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1216)
> at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1249)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.supervisor.main(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1710) java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
[
https://issues.apache.org/jira/browse/STORM-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241162#comment-15241162
]
Robert Joseph Evans commented on STORM-1710:
The issue is that with storm 1.0 we moved from backtype.storm to
org.apache.storm. This is not a rolling upgrade so if you have any state in
zookeeper or stored on the local disk for your cluster you need to delete it,
and also update any configs that you have that are pointing to backtype.storm
classes.
> java.lang.ClassNotFoundException: backtype.storm.generated.LSSupervisorId
> -
>
> Key: STORM-1710
> URL: https://issues.apache.org/jira/browse/STORM-1710
> Project: Apache Storm
> Issue Type: Question
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Linux 2.6.32-358.el6.x86_64 #1 SMP Tue Jan 29 11:47:41
> EST 2013 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Guangxin Zhang
> Fix For: 1.0.0
>
>
> 2016-04-14 09:07:22.226 o.a.s.d.supervisor [ERROR] Error on initialization of
> server mk-supervisor
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> backtype.storm.generated.LSSupervisorId
> at org.apache.storm.utils.LocalState.deserialize(LocalState.java:83)
> at org.apache.storm.utils.LocalState.get(LocalState.java:130)
> at
> org.apache.storm.local_state$ls_supervisor_id.invoke(local_state.clj:61)
> at
> org.apache.storm.daemon.supervisor$standalone_supervisor$reify__9646.prepare(supervisor.clj:1228)
> at
> org.apache.storm.daemon.supervisor$fn__9503$exec_fn__1826__auto9504.invoke(supervisor.clj:781)
> at clojure.lang.AFn.applyToHelper(AFn.java:160)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:630)
> at
> org.apache.storm.daemon.supervisor$fn__9503$mk_supervisor__9548.doInvoke(supervisor.clj:779)
> at clojure.lang.RestFn.invoke(RestFn.java:436)
> at
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1216)
> at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1249)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.supervisor.main(Unknown Source)
> Caused by: java.lang.ClassNotFoundException:
> backtype.storm.generated.LSSupervisorId
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:186)
> at org.apache.storm.utils.LocalState.deserialize(LocalState.java:78)
> ... 14 more
> 2016-04-14 09:07:22.239 o.a.s.util [ERROR] Halting process: ("Error on
> initialization")
> java.lang.RuntimeException: ("Error on initialization")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at
> org.apache.storm.daemon.supervisor$fn__9503$mk_supervisor__9548.doInvoke(supervisor.clj:779)
> at clojure.lang.RestFn.invoke(RestFn.java:436)
> at
> org.apache.storm.daemon.supervisor$_launch.invoke(supervisor.clj:1216)
> at org.apache.storm.daemon.supervisor$_main.invoke(supervisor.clj:1249)
> at clojure.lang.AFn.applyToHelper(AFn.java:152)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at org.apache.storm.daemon.supervisor.main(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (STORM-1658) documents improvements and links fixes
[ https://issues.apache.org/jira/browse/STORM-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1658. Resolution: Fixed Fix Version/s: (was: 0.9.6) 2.0.0 Thanks [~vesense] and [~kabhwan], I think I got them all merged into the correct places now, and the webpage updated. If you see any issues please let me know. > documents improvements and links fixes > -- > > Key: STORM-1658 > URL: https://issues.apache.org/jira/browse/STORM-1658 > Project: Apache Storm > Issue Type: Bug > Components: documentation >Reporter: Xin Wang >Assignee: Xin Wang > Fix For: 2.0.0 > > > Direct groupings(_Direct-groupings.html_) in releases > 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT,0.10.0 is not found(404). > Deamon Metrics/Monitoring(_storm-metrics-profiling-internal-actions.html_) in > releases 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT is not found(404). > For the former, I cannot find _Direct-groupings.md_ in docs. For the latter, > I find _storm-metrics-profiling-internal-actions.md_ in docs dir. > According to Brian‘s report in storm-user mailing list: > (1) http://storm.apache.org/ > at the bottom has a link to "Tutorial": > http://storm.apache.org/releases/current/tutorial.html > but this gives a 404 > Either the link needs to change to > http://storm.apache.org/releases/current/Tutorial.html > or the page itself needs to be renamed to "tutorial.html" > (2) http://storm.apache.org/releases/0.10.0/index.html > Under "Intermediate" > * The "Direct Groupings" link points to > http://storm.apache.org/releases/0.10.0/Direct-groupings.html > which is non-existent. > * The "Lifecycle of a trident tuple" link just points to the documentation > index, > http://storm.apache.org/releases/0.10.0/index.html > (3) There are many pages with broken images. For example > * > http://storm.apache.org/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.html > has broken image pointing to > http://storm.apache.org/releases/0.10.0/images/relationships-worker-processes-executors-tasks.png > http://storm.apache.org/releases/0.10.0/images/example-of-a-running-topology.png > (these give 404 errors) > * http://storm.apache.org/releases/0.10.0/Trident-tutorial.html > has a broken image pointing to > http://storm.apache.org/releases/0.10.0/images/batched-stream.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (STORM-1658) documents improvements and links fixes
[ https://issues.apache.org/jira/browse/STORM-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reopened STORM-1658: [~kabhwan], Sorry I missed them > documents improvements and links fixes > -- > > Key: STORM-1658 > URL: https://issues.apache.org/jira/browse/STORM-1658 > Project: Apache Storm > Issue Type: Bug > Components: documentation >Reporter: Xin Wang >Assignee: Xin Wang > Fix For: 0.9.6 > > > Direct groupings(_Direct-groupings.html_) in releases > 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT,0.10.0 is not found(404). > Deamon Metrics/Monitoring(_storm-metrics-profiling-internal-actions.html_) in > releases 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT is not found(404). > For the former, I cannot find _Direct-groupings.md_ in docs. For the latter, > I find _storm-metrics-profiling-internal-actions.md_ in docs dir. > According to Brian‘s report in storm-user mailing list: > (1) http://storm.apache.org/ > at the bottom has a link to "Tutorial": > http://storm.apache.org/releases/current/tutorial.html > but this gives a 404 > Either the link needs to change to > http://storm.apache.org/releases/current/Tutorial.html > or the page itself needs to be renamed to "tutorial.html" > (2) http://storm.apache.org/releases/0.10.0/index.html > Under "Intermediate" > * The "Direct Groupings" link points to > http://storm.apache.org/releases/0.10.0/Direct-groupings.html > which is non-existent. > * The "Lifecycle of a trident tuple" link just points to the documentation > index, > http://storm.apache.org/releases/0.10.0/index.html > (3) There are many pages with broken images. For example > * > http://storm.apache.org/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.html > has broken image pointing to > http://storm.apache.org/releases/0.10.0/images/relationships-worker-processes-executors-tasks.png > http://storm.apache.org/releases/0.10.0/images/example-of-a-running-topology.png > (these give 404 errors) > * http://storm.apache.org/releases/0.10.0/Trident-tutorial.html > has a broken image pointing to > http://storm.apache.org/releases/0.10.0/images/batched-stream.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1658) documents improvements and links fixes
[ https://issues.apache.org/jira/browse/STORM-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1658. Resolution: Fixed Fix Version/s: 0.9.6 Thanks [~vesense], I merged this into 0.9.x-branch and pushed the changed .md and generated .html changes to subversion so they should show up for the 0.9.6 release docs. > documents improvements and links fixes > -- > > Key: STORM-1658 > URL: https://issues.apache.org/jira/browse/STORM-1658 > Project: Apache Storm > Issue Type: Bug > Components: documentation >Reporter: Xin Wang >Assignee: Xin Wang > Fix For: 0.9.6 > > > Direct groupings(_Direct-groupings.html_) in releases > 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT,0.10.0 is not found(404). > Deamon Metrics/Monitoring(_storm-metrics-profiling-internal-actions.html_) in > releases 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT is not found(404). > For the former, I cannot find _Direct-groupings.md_ in docs. For the latter, > I find _storm-metrics-profiling-internal-actions.md_ in docs dir. > According to Brian‘s report in storm-user mailing list: > (1) http://storm.apache.org/ > at the bottom has a link to "Tutorial": > http://storm.apache.org/releases/current/tutorial.html > but this gives a 404 > Either the link needs to change to > http://storm.apache.org/releases/current/Tutorial.html > or the page itself needs to be renamed to "tutorial.html" > (2) http://storm.apache.org/releases/0.10.0/index.html > Under "Intermediate" > * The "Direct Groupings" link points to > http://storm.apache.org/releases/0.10.0/Direct-groupings.html > which is non-existent. > * The "Lifecycle of a trident tuple" link just points to the documentation > index, > http://storm.apache.org/releases/0.10.0/index.html > (3) There are many pages with broken images. For example > * > http://storm.apache.org/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.html > has broken image pointing to > http://storm.apache.org/releases/0.10.0/images/relationships-worker-processes-executors-tasks.png > http://storm.apache.org/releases/0.10.0/images/example-of-a-running-topology.png > (these give 404 errors) > * http://storm.apache.org/releases/0.10.0/Trident-tutorial.html > has a broken image pointing to > http://storm.apache.org/releases/0.10.0/images/batched-stream.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1687) Divide by zero exception in stats
[
https://issues.apache.org/jira/browse/STORM-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-1687.
Resolution: Fixed
Thanks [~zhuoliu],
I merged this into master and 1.x-branch
> Divide by zero exception in stats
> -
>
> Key: STORM-1687
> URL: https://issues.apache.org/jira/browse/STORM-1687
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
> Fix For: 1.0.0, 2.0.0
>
>
> Since uptime can be 0, this will cause ArithmeticException: Divide by zero in
> compute-agg-capacity.
> This will happen for both stats.clj in 1.x and StatsUtil.java in master (2.0).
> {noformat}
> java.lang.ArithmeticException: Divide by zero
> at clojure.lang.Numbers.divide(Numbers.java:156)
> at clojure.core$SLASH.invoke(core.clj:986)
> at clojure.lang.AFn.applyToHelper(AFn.java:156)
> at clojure.lang.RestFn.applyTo(RestFn.java:132)
> at clojure.core$apply.invoke(core.clj:626)
> at backtype.storm.util$div.doInvoke(util.clj:355)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at backtype.storm.stats$compute_agg_capacity$fn__2249.invoke(stats.clj:409)
> at backtype.storm.stats$compute_agg_capacity.invoke(stats.clj:404)
> at backtype.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:555)
> at backtype.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:724)
> at backtype.storm.stats$fn__2319.invoke(stats.clj:772)
> at clojure.lang.MultiFn.invoke(MultiFn.java:241)
> at clojure.lang.AFn.applyToHelper(AFn.java:165)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:628)
> at clojure.core$partial$fn__4230.doInvoke(core.clj:2470)
> at clojure.lang.RestFn.invoke(RestFn.java:421)
> at clojure.core.protocols$fn__6086.invoke(protocols.clj:143)
> at clojure.core.protocols$fn_6057$G6052_6066.invoke(protocols.clj:19)
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> at clojure.core.protocols$fn__6078.invoke(protocols.clj:54)
> at clojure.core.protocols$fn_6031$G6026_6044.invoke(protocols.clj:13)
> at clojure.core$reduce.invoke(core.clj:6289)
> at backtype.storm.stats$aggregate_topo_stats.invoke(stats.clj:854)
> at backtype.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1008)
> at
> backtype.storm.daemon.nimbus$fn_5838$exec_fn1478auto$reify_5862.getTopologyPageInfo(nimbus.clj:1729)
> at
> backtype.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3651)
> at
> backtype.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3635)
> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> backtype.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:143)
> at
> org.apache.thrift7.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (STORM-1691) Earlier storm documentation links are broken.
[ https://issues.apache.org/jira/browse/STORM-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1691. Resolution: Fixed [~satish.duggana], I just checked in a fix for this. Please check your bookmarks to be sure that they are redirecting to what you would expect. You can reopen this if something is still not working like you would want. > Earlier storm documentation links are broken. > - > > Key: STORM-1691 > URL: https://issues.apache.org/jira/browse/STORM-1691 > Project: Apache Storm > Issue Type: Bug >Reporter: Satish Duggana >Assignee: Robert Joseph Evans > > Earlier Storm documentation links are broken. Whoever has bookmarked earlier > links get 404 now. > These should be redirected to the latest release version docs instead of > completely removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1691) Earlier storm documentation links are broken.
[ https://issues.apache.org/jira/browse/STORM-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans reassigned STORM-1691: -- Assignee: Robert Joseph Evans > Earlier storm documentation links are broken. > - > > Key: STORM-1691 > URL: https://issues.apache.org/jira/browse/STORM-1691 > Project: Apache Storm > Issue Type: Bug >Reporter: Satish Duggana >Assignee: Robert Joseph Evans > > Earlier Storm documentation links are broken. Whoever has bookmarked earlier > links get 404 now. > These should be redirected to the latest release version docs instead of > completely removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1279) port backtype.storm.daemon.supervisor to java
[ https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1279. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~Johnbaba], I merged this into master. Keep up the good work. > port backtype.storm.daemon.supervisor to java > - > > Key: STORM-1279 > URL: https://issues.apache.org/jira/browse/STORM-1279 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > Attachments: Discussion about supervisor.pdf > > > https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor > as an example > backtype.storm.event usage should be replaced with built-in java threadpools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1271) port backtype.storm.daemon.task to java
[ https://issues.apache.org/jira/browse/STORM-1271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1271. Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~abhishek.agarwal], I merged this into master. > port backtype.storm.daemon.task to java > --- > > Key: STORM-1271 > URL: https://issues.apache.org/jira/browse/STORM-1271 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Abhishek Agarwal > Labels: java-migration, jstorm-merger > Fix For: 2.0.0 > > > helper functions for task data and sending tuples -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1672) Stats not get class cast exception
Robert Joseph Evans created STORM-1672:
--
Summary: Stats not get class cast exception
Key: STORM-1672
URL: https://issues.apache.org/jira/browse/STORM-1672
Project: Apache Storm
Issue Type: Bug
Components: storm-core
Affects Versions: 2.0.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Priority: Critical
Component page in UI
{code}
2016-03-31 14:21:44.576 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer [ERROR]
Unexpected throwable while invoking!
java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.Map
at
org.apache.storm.stats.StatsUtil.filterSysStreams(StatsUtil.java:1696)
at
org.apache.storm.stats.StatsUtil.aggPreMergeCompPageBolt(StatsUtil.java:240)
at
org.apache.storm.stats.StatsUtil.aggCompExecStats(StatsUtil.java:1130)
at
org.apache.storm.stats.StatsUtil.aggregateCompStats(StatsUtil.java:1108)
at
org.apache.storm.stats.StatsUtil.aggCompExecsStats(StatsUtil.java:1236)
at
org.apache.storm.daemon.nimbus$fn__3490$exec_fn__789__auto__$reify__3519.getComponentPageInfo(nimbus.clj:2130)
at
org.apache.storm.generated.Nimbus$Processor$getComponentPageInfo.getResult(Nimbus.java:3826)
at
org.apache.storm.generated.Nimbus$Processor$getComponentPageInfo.getResult(Nimbus.java:3810)
at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
at
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated STORM-822: -- Fix Version/s: 2.0.0 > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1666) Kill from the UI fails silently.
[
https://issues.apache.org/jira/browse/STORM-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-1666.
Resolution: Fixed
Assignee: Robert Joseph Evans
Fix Version/s: 2.0.0
Went in with STORM-1663
> Kill from the UI fails silently.
>
>
> Key: STORM-1666
> URL: https://issues.apache.org/jira/browse/STORM-1666
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Minor
> Fix For: 2.0.0
>
>
> {code}
> 2016-03-30 14:02:21.613 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer
> [ERROR] Unexpected throwable while invoking!
> java.lang.NullPointerException
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:26)
> at
> org.apache.storm.daemon.nimbus$fn__3070$exec_fn__2175__auto__$reify__3099$iter__3210__3214$fn__3215.invoke(nimbus.clj:1888)
> at clojure.lang.LazySeq.sval(LazySeq.java:40)
> at clojure.lang.LazySeq.seq(LazySeq.java:49)
> at clojure.lang.RT.seq(RT.java:507)
> at clojure.core$seq__4128.invoke(core.clj:137)
> at clojure.core$dorun.invoke(core.clj:3009)
> at clojure.core$doall.invoke(core.clj:3025)
> at
> org.apache.storm.daemon.nimbus$fn__3070$exec_fn__2175__auto__$reify__3099.getTopologyInfoWithOpts(nimbus.clj:1885)
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfoWithOpts.getResult(Nimbus.java:3774)
> at
> org.apache.storm.generated.Nimbus$Processor$getTopologyInfoWithOpts.getResult(Nimbus.java:3758)
> at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
> at
> org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (STORM-1665) Worker cannot instantiate kryo
[
https://issues.apache.org/jira/browse/STORM-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved STORM-1665.
Resolution: Fixed
Went in with STORM-1663
> Worker cannot instantiate kryo
> --
>
> Key: STORM-1665
> URL: https://issues.apache.org/jira/browse/STORM-1665
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Critical
> Fix For: 2.0.0
>
>
> {code}
> java.lang.NoClassDefFoundError: org/objenesis/strategy/InstantiatorStrategy
> at
> org.apache.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:47)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.serialization.KryoValuesSerializer.(KryoValuesSerializer.java:33)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.messaging.netty.Server.(Server.java:71)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.messaging.netty.Context.bind(Context.java:67)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.worker$worker_data$fn__1953.invoke(worker.clj:280)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.util$assoc_apply_self.invoke(util.clj:201)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at org.apache.storm.daemon.worker$worker_data.invoke(worker.clj:277)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at
> org.apache.storm.daemon.worker$fn__2256$exec_fn__784__auto__$reify__2258.run(worker.clj:652)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0]
> at
> org.apache.storm.daemon.worker$fn__2256$exec_fn__784__auto2257.invoke(worker.clj:650)
> ~[storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
> at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
> at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__2256$mk_worker__2353.doInvoke(worker.clj:624)
> [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$_main.invoke(worker.clj:821)
> [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker.main(Unknown Source)
> [storm-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
> Caused by: java.lang.ClassNotFoundException:
> org.objenesis.strategy.InstantiatorStrategy
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372) ~[?:1.8.0]
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[?:1.8.0]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0]
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> ~[?:1.8.0]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> ~[?:1.8.0]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0]
> ... 21 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
