[
https://issues.apache.org/jira/browse/STORM-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311925#comment-14311925
]
Li Chaomo edited comment on STORM-605 at 2/9/15 8:36 AM:
---------------------------------------------------------
[~sriharsha] hello, i found something today though i don't know why it was
here is the thing , we use clojure dsl under storm . The bolt always loop died
when it run the prepare function,So i guess
maybe something wrong happen when load the clojure code. Then i build a storm
from github mirror and log the exception when storm load the clojure bolt
backtype.storm.utils/Utils
public static IFn loadClojureFn(String namespace, String name) {
try {
clojure.lang.Compiler.eval(RT.readString("(require '" + namespace +
")"));
} catch (Exception e) {
//if playing from the repl and defining functions, file won't exist
LOG.error("读取clojure bolt代码异常 namespace=" + namespace + ", name=" + name, e);
}
return (IFn) RT.var(namespace, name).deref();
}
here is the log when the bolt throw exception
2015-02-09 16:19:02 b.s.u.Utils [ERROR] 读取clojure bolt代码异常
namespace=com.pajk.risk.storm.clj.engine, name=count-bolt__
clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to
resolve symbol: conditionCache in this context,
compiling:(com/pajk/risk/storm/clj/calc.clj:12:13)
at clojure.lang.Compiler.analyze(Compiler.java:6464)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyze(Compiler.java:6406)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler$HostExpr$Parser.parse(Compiler.java:924)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyze(Compiler.java:6445)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6632)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyze(Compiler.java:6445)
~[clojure-1.6.0.jar:na]
but i do define the conditionCache in a another clojure file, and i load it by
:use , here is my code
(ns com.pajk.risk.storm.clj.calc
(:import (java.util Map)
(com.pajk.risk.engine.api.model UniformEvent))
(:use com.pajk.risk.storm.clj.cache)
(:use com.pajk.risk.storm.clj.mock)
(:use com.pajk.risk.storm.clj.util)
(:require [clojure.tools.logging :as log]))
conditionCache is defined in com.pajk.risk.storm.clj.cache
(def ^LoadingCache conditionCache (-> (CacheBuilder/newBuilder)
(.maximumSize 10000)
(.build (proxy [CacheLoader] [] (load
[^String key]
(AviatorEvaluator/compile key))))))
My question is how the storm load my clojure bolt and how to load its
dependency , Thx
was (Author: newday1):
[~sriharsha] hello, i found something today though i don't know why it was
here is the thing , we use clojure dsl under storm . The bolt always loop died
when it run the prepare function,So i guess
maybe something wrong happen when load the clojure code. Then i build a storm
from github mirror and log the exception when storm load the clojure bolt
backtype.storm.utils/Utils
public static IFn loadClojureFn(String namespace, String name) {
try {
clojure.lang.Compiler.eval(RT.readString("(require '" + namespace +
")"));
} catch (Exception e) {
//if playing from the repl and defining functions, file won't exist
LOG.error("读取clojure bolt代码异常 namespace=" + namespace + ", name=" +
name, e);
}
return (IFn) RT.var(namespace, name).deref();
}
here is the log when the bolt throw exception
2015-02-09 16:19:02 b.s.u.Utils [ERROR] 读取clojure bolt代码异常
namespace=com.pajk.risk.storm.clj.engine, name=count-bolt__
clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Unable to
resolve symbol: conditionCache in this context,
compiling:(com/pajk/risk/storm/clj/calc.clj:12:13)
at clojure.lang.Compiler.analyze(Compiler.java:6464)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyze(Compiler.java:6406)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler$HostExpr$Parser.parse(Compiler.java:924)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6644)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyze(Compiler.java:6445)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyzeSeq(Compiler.java:6632)
~[clojure-1.6.0.jar:na]
at clojure.lang.Compiler.analyze(Compiler.java:6445)
~[clojure-1.6.0.jar:na]
but i do define the conditionCache in a another clojure file, and i load it by
:use , here is my code
(ns com.pajk.risk.storm.clj.calc
(:import (java.util Map)
(com.pajk.risk.engine.api.model UniformEvent))
(:use com.pajk.risk.storm.clj.cache)
(:use com.pajk.risk.storm.clj.mock)
(:use com.pajk.risk.storm.clj.util)
(:require [clojure.tools.logging :as log]))
conditionCache is defined in com.pajk.risk.storm.clj.cache
(def ^LoadingCache conditionCache (-> (CacheBuilder/newBuilder)
(.maximumSize 10000)
(.build (proxy [CacheLoader] [] (load
[^String key]
(AviatorEvaluator/compile key))))))
My question is how the storm load my clojure bolt and how to load its
dependency , Thx
> Attempting to call unbound fn during bolt prepare
> -------------------------------------------------
>
> Key: STORM-605
> URL: https://issues.apache.org/jira/browse/STORM-605
> Project: Apache Storm
> Issue Type: Bug
> Affects Versions: 0.9.3
> Reporter: Philippe Guillebert
>
> We had a bunch of topologies running very well under Storm 0.8.2 until last
> week when we switched to storm 0.9.2-incubating. We use the clojure DSL,
> and clojure 1.5.1 (only).
> Since the change, we have a large topology (about 30 bolts, parallellism=10
> or 20 per bolt, total 372 tasks on 10 workers) that fails on startup with
> several bolts showing the exception :
> java.lang.RuntimeException: java.lang.IllegalStateException: Attempting to
> call unbound fn: #'entry-dedup.bolt/dedup__ at
> backtype.storm.clojure.ClojureBolt.prepare(ClojureBolt.java:77) ...
> This can occur on one or several bolts at random and is not consistent
> between restarts.
> The topology is indeed quite long to initialize (a dozen seconds) due to
> several models being loaded but this was OK in 0.8.2.
> Another (shorter) topology works most of the time but shows this behaviour
> on some restarts sometimes.
> We found a workaround that works most of the time : start the topology in
> the INACTIVE state, then wait 200 seconds, then activate it. But this
> doesn't really solve our problem because sometimes Storm tries to rebalance
> the topologies by itself and reassigns the topology without our little trick,
> effectively crashing them.
> The same behavior is present with storm 0.9.3.
> So maybe something changed in storm that introduces a kind of race
> condition during initializaion of some bolts on larger topologies ? Maybe
> this is a consequence to the switch to Netty ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
