[GitHub] storm pull request: Storm-166: Nimbus HA design doc and implementa...
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/354#issuecomment-76831122 @ptgoetz @revans2 @Parth-Brahmbhatt I am +1 on the patch as well. I like the idea of 0.11-branch makes it easier for anyone testing it too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-554) the get-task-object function in task.clj,the type of first param topology should be ^StormTopology not ^TopologyContext
[ https://issues.apache.org/jira/browse/STORM-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344288#comment-14344288 ] ASF GitHub Bot commented on STORM-554: -- Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/433#issuecomment-76870913 @d2r I have added [STORM-554] to the title of this pull request and closed #312. the get-task-object function in task.clj,the type of first param topology should be ^StormTopology not ^TopologyContext --- Key: STORM-554 URL: https://issues.apache.org/jira/browse/STORM-554 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3-rc2 Reporter: zhangjinlong Assignee: zhangjinlong Fix For: 0.9.3-rc2 the get-task-object function in task.clj,the type of first param topology should be ^StormTopology not ^TopologyContext -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Storm Static and Dynamic views
Hi , I am student from CMU and I am interested in understanding about the Storms static and dynamic architecture. I want to understand what are all the modules present and how they connect to each other. As part an academic project i am interested window management framework of storm and wish to understand how much of it there in storm already and which part of storm corresponds to that area. Please share any document on the design architecture at module level if any. Thanks, Pradheep CMU
[GitHub] storm pull request: Update get-task-object function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/312#issuecomment-76868454 @d2r ok, i have closed this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-693) KafkaBolt exception handling improvement
[ https://issues.apache.org/jira/browse/STORM-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344095#comment-14344095 ] ASF GitHub Bot commented on STORM-693: -- GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/storm/pull/453 STORM-693: when bolt fails to write tuple, it should report error instea... ...d of silently acking. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/incubator-storm STORM-693 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/453.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #453 commit aadccb4cb2a6602651d31542ecf2c96d72a0edfa Author: Parth Brahmbhatt brahmbhatt.pa...@gmail.com Date: 2015-03-03T00:06:58Z STORM-693: when bolt fails to write tuple, it should report error instead of silently acking. KafkaBolt exception handling improvement Key: STORM-693 URL: https://issues.apache.org/jira/browse/STORM-693 Project: Apache Storm Issue Type: Bug Components: storm-kafka Affects Versions: 0.9.3 Reporter: Rick Kellogg Within the KafkaBolt execute method, an error message is logged if any sort of error occurs communicating with Kafka. Unfortunately the input is still acknowledged. Upon review of the HdfsBolt HiveBolt, I believe the exception handling block should include the following two lines: this.collector.reportError(ex); this.collector.fail(input); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: Update get-task-object function, change the ...
Github user BuDongDong commented on the pull request: https://github.com/apache/storm/pull/433#issuecomment-76870913 @d2r I have added [STORM-554] to the title of this pull request and closed #312. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[
https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344522#comment-14344522
]
ASF GitHub Bot commented on STORM-682:
--
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/437#issuecomment-76887513
+1. @nathanmarz can you review the latest patch. Thanks.
Supervisor local worker state corrupted and failing to start.
-
Key: STORM-682
URL: https://issues.apache.org/jira/browse/STORM-682
Project: Apache Storm
Issue Type: Bug
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt
If supervisor's cleanup of a worker fails to delete some heartbeat files the
local state of the supervisors get corrupted.The only way to recover the
supervisor from this state is to delete the local state folder where
supervisor stores all worker information.This fix can get very cumbersome if
it happens on multiple worker nodes.
The root cause of the issue is the order in which worker heartbeat versioned
store files are created vs the deletion order of those files. LocalState.put
first creates a data file X and then marks a success by creating a file
X.version. During get it first checks for all *.version files , tries to
find the largest value of X and then issues a read against X. See the below
pseudo code
{code:java}
start_supervisor() {
workerIds = `ls local-state/workers`
for each workerId in workerIds
versions = `ls local-state/workers/workerId/heartbeats/*.version`
latest_version = max(versions)
read local-state/workers/workerId/heartbeats/latest_version [Note there
is no .version extension]
}
{code}
During cleanup it first tries to delete file X and then X.version. If X gets
deleted but X.version fails to delete the supervisor fails to start with
FileNotFoundException in the code above.
We propose to change the deletion order so the .version files get deleted
before the data file and catch any IOException when reading worker heartbeats
to avoid supervisor failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Storm REST API returns 404 when retrieving stats for components whose name includes commas?
I observed that components whose ID includes a comma cannot be queried
for:
# Using the encoded name (`,` is encoded to `%2C`)
$ curl -s
$API_BASE/api/v1/topology/MyTopology/component/MyBolt%2CVariantA
Page not found
# Using the raw name (`,` is left as is)
$ curl -s
$API_BASE/api/v1/topology/MyTopology/component/MyBolt,VariantA
Page not found
Should I use a different URL to query the component stats or is this a
legitimate bug? If the latter, then I'll file a bug report.
FWIW the REST API does correctly return the component ID when querying
for, say, the topology information:
$ curl -s $API_BASE/api/v1/topology/MyTopology
{
...
bolts: [
{
boltId: MyBolt,VariantA,
encodedBoltId: MyBolt%2CVariantA,
}
]
...
}
--Michael
[jira] [Commented] (STORM-695) storm CLI tool reports zero exit code on error scenario, take 2
[
https://issues.apache.org/jira/browse/STORM-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342955#comment-14342955
]
Kai Sasaki commented on STORM-695:
--
[~miguno] CLI tools cannot handle any errors with current command tool. So I
think it will be needed. But I have a question. Is there anyway to handle
exceptions and errors through thrift interface? Though I am not familiar with
thrift so much, I want to know how to do it. Thank you.
storm CLI tool reports zero exit code on error scenario, take 2
---
Key: STORM-695
URL: https://issues.apache.org/jira/browse/STORM-695
Project: Apache Storm
Issue Type: Bug
Affects Versions: 0.9.3
Reporter: Michael Noll
Priority: Minor
Commands such as storm kill non-existing-topology will return an exit code
of zero, indicating success when in fact the command failed.
h3. How to reproduce
Here is but one example where the {{storm}} CLI tool violates shell best
practices:
{code}
# Let's kill a topology that is in fact not running in the cluster.
$ storm kill i-do-not-exist-topo
snip
Exception in thread main NotAliveException(msg:i-do-not-exist-topo is not
alive)
# Print the exit code of last command.
$ echo $?
0 # but since the kill command failed this should be non-zero!
{code}
Another example is the storm jar command. If you attempt to submit a
topology that has the same name as an existing, running topology, the storm
jar command will not submit the topology -- instead it will print an
exception (think: the topology FooName is already running), which is ok,
but it will then exit with a return code of zero, which indicates success
(which is wrong).
h3. Impact
This bug prevents automated deployment tools such as Ansible or Puppet as
well as ad-hoc CLI scripting (think: fire-fighting Ops teams) to work
properly because Storm violates shell conventions by not returning non-zero
exit codes in case of failures.
h3. How to fix
From what I understand the solution is two-fold:
# The various Storm commands that are being called by the {{storm}} script
must return proper exit codes.
# The {{storm}} script must store these exit codes and return itself with the
respective exit code of the Storm command it actually ran.
For example, here's the current code that implements the storm kill command:
{code}
# In: bin/storm
def kill(*args):
Syntax: [storm kill topology-name [-w wait-time-secs]]
Kills the topology with the name topology-name. Storm will
first deactivate the topology's spouts for the duration of
the topology's message timeout to allow all messages currently
being processed to finish processing. Storm will then shutdown
the workers and clean up their state. You can override the length
of time Storm waits between deactivation and shutdown with the -w flag.
exec_storm_class(
backtype.storm.command.kill_topology,
args=args,
jvmtype=-client,
extrajars=[USER_CONF_DIR, STORM_BIN_DIR])
{code}
which in turn calls the following code in {{kill_topology.clj}}:
{code}
;; In: backtype.storm.command.kill-topology
(ns backtype.storm.command.kill-topology
(:use [clojure.tools.cli :only [cli]])
(:use [backtype.storm thrift config log])
(:import [backtype.storm.generated KillOptions])
(:gen-class))
(defn -main [ args]
(let [[{wait :wait} [name] _] (cli args [-w --wait :default nil
:parse-fn #(Integer/parseInt %)])
opts (KillOptions.)]
(if wait (.set_wait_secs opts wait))
(with-configured-nimbus-connection nimbus
(.killTopologyWithOpts nimbus name opts)
(log-message Killed topology: name)
)))
{code}
which in turn calls the following code in {{nimbus.clj}}:
{code}
;; In: backtype.storm.daemon.nimbus
(^void killTopologyWithOpts [this ^String storm-name ^KillOptions
options]
(check-storm-active! nimbus storm-name true)
(let [topology-conf (try-read-storm-conf-from-name conf storm-name
nimbus)]
(check-authorization! nimbus storm-name topology-conf
killTopology))
(let [wait-amt (if (.is_set_wait_secs options)
(.get_wait_secs options)
)]
(transition-name! nimbus storm-name [:kill wait-amt] true)
))
{code}
As you can see the current implementation does not pass success/failure
information back to the caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Storm UI Issue
HI ALL , Kindly help me to sort out out the issue with storm UI. In our topology , we have a Kafka spout and 2 bolts (which doesn't emit any downstream tuples). My bolts jsut process the incoming tuples .And executors to Kafka spout and bolts are set to one . When running our topology , However my bolts are not emitting any downstream tuples,but still i am able to see the Emitted and Transferred values in bolts section .I hope this not the expected behaviour . Kndly any one help me . Please let me know ,in case of more information needed . Regards , Nanda .
[GitHub] storm pull request: Update get-task-object function, change the ...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/312#discussion_r25610339 --- Diff: storm-core/src/clj/backtype/storm/stats.clj --- @@ -67,7 +67,7 @@ (defrecord RollingWindowSet [updater extractor windows all-time]) (defn rolling-window-set [updater merger extractor num-buckets bucket-sizes] - (RollingWindowSet. updater extractor (dofor [s bucket-sizes] (rolling-window updater merger extractor s num-buckets)) nil) + (RollingWindowSet. updater extractor (dofor [s bucket-sizes] (rolling-window updater merger extractor num-buckets s)) nil) --- End diff -- This breaks the call to `rolling-window`. Please revert. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update get-task-object function, change the ...
Github user d2r commented on a diff in the pull request: https://github.com/apache/storm/pull/312#discussion_r25610832 --- Diff: storm-core/src/clj/backtype/storm/stats.clj --- @@ -67,7 +67,7 @@ (defrecord RollingWindowSet [updater extractor windows all-time]) (defn rolling-window-set [updater merger extractor num-buckets bucket-sizes] - (RollingWindowSet. updater extractor (dofor [s bucket-sizes] (rolling-window updater merger extractor s num-buckets)) nil) + (RollingWindowSet. updater extractor (dofor [s bucket-sizes] (rolling-window updater merger extractor num-buckets s)) nil) --- End diff -- Oh, never mind, I should be looking at #433. @BuDongDong , thanks for the update. Would you close this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Storm REST API returns 404 when retrieving stats for components whose name includes commas?
This feels like a bug in the RESTful API to me
- Bobby
On Monday, March 2, 2015 7:48 AM, Michael G. Noll
mich...@michael-noll.com wrote:
I observed that components whose ID includes a comma cannot be queried
for:
# Using the encoded name (`,` is encoded to `%2C`)
$ curl -s
$API_BASE/api/v1/topology/MyTopology/component/MyBolt%2CVariantA
Page not found
# Using the raw name (`,` is left as is)
$ curl -s
$API_BASE/api/v1/topology/MyTopology/component/MyBolt,VariantA
Page not found
Should I use a different URL to query the component stats or is this a
legitimate bug? If the latter, then I'll file a bug report.
FWIW the REST API does correctly return the component ID when querying
for, say, the topology information:
$ curl -s $API_BASE/api/v1/topology/MyTopology
{
...
bolts: [
{
boltId: MyBolt,VariantA,
encodedBoltId: MyBolt%2CVariantA,
}
]
...
}
--Michael
Re: Storm UI Issue
Do you have system stats enabled? The numbers you see are aggregates across all of the bolts, and all of the other things they could be sending messages to. If you have system stats enabled on the UI you could be seeing metrics or ack messages being sent around. It is also possible that storm has a bug in it that we are not filtering some system stats properly. - Bobby On Monday, March 2, 2015 3:36 AM, nanda n nanda.ja...@gmail.com wrote: HI ALL , Kindly help me to sort out out the issue with storm UI. In our topology , we have a Kafka spout and 2 bolts (which doesn't emit any downstream tuples). My bolts jsut process the incoming tuples .And executors to Kafka spout and bolts are set to one . When running our topology , However my bolts are not emitting any downstream tuples,but still i am able to see the Emitted and Transferred values in bolts section .I hope this not the expected behaviour . Kndly any one help me . Please let me know ,in case of more information needed . Regards , Nanda .
[jira] [Commented] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343375#comment-14343375 ] ASF GitHub Bot commented on STORM-557: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/314 High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343374#comment-14343374 ] ASF GitHub Bot commented on STORM-557: -- Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-76749927 +1 High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-682: supervisor should handle worker sta...
Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-76751433 Looks good. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Update get-task-object function, change the ...
Github user d2r commented on the pull request: https://github.com/apache/storm/pull/433#issuecomment-76752033 +1 Yes, it appears the type hint is wrong. This works anyway because the method calls in `get-task-object` are correct for the actual class (`StormTopology`) of the instance. But we should correct the type hint. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishor Patil updated STORM-557: --- Fix Version/s: 0.10.0 High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.10.0 Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-532) Supervisor should restart worker immediately, if the worker process does not exist any more
[ https://issues.apache.org/jira/browse/STORM-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343345#comment-14343345 ] ASF GitHub Bot commented on STORM-532: -- Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/296#issuecomment-76745378 @caofangkun, could you please upmerge this change? Supervisor should restart worker immediately, if the worker process does not exist any more Key: STORM-532 URL: https://issues.apache.org/jira/browse/STORM-532 Project: Apache Storm Issue Type: Improvement Affects Versions: 0.10.0 Reporter: caofangkun Assignee: caofangkun Priority: Minor For now if the worker process does not exist any more Supervisor will have to wait a few seconds for worker heartbeart timeout and restart worker . If supervisor knows the worker processid and check if the process exists in the sync-processes thread ,may need less time to restart worker. 1: record worker process id in the worker local heartbeart 2: in supervisor sync-processes ,get process id from worker local heartbeat and check if the process exits 3: if not restart it immediately -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-532:Supervisor should restart worker imm...
Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/296#issuecomment-76745378 @caofangkun, could you please upmerge this change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-446) secure Impersonation in storm
[ https://issues.apache.org/jira/browse/STORM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343356#comment-14343356 ] ASF GitHub Bot commented on STORM-446: -- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/448#discussion_r25613390 --- Diff: storm-core/src/clj/backtype/storm/ui/core.clj --- @@ -45,28 +45,46 @@ (def ^:dynamic *STORM-CONF* (read-storm-config)) (def ^:dynamic *UI-ACL-HANDLER* (mk-authorization-handler (*STORM-CONF* NIMBUS-AUTHORIZER) *STORM-CONF*)) +(def ^:dynamic *UI-IMPERSONATION-HANDLER* (mk-authorization-handler (*STORM-CONF* NIMBUS-IMPERSONATION-AUTHORIZER) *STORM-CONF*)) (def http-creds-handler (AuthUtils/GetUiHttpCredentialsPlugin *STORM-CONF*)) (defmacro with-nimbus [nimbus-sym body] - `(thrift/with-nimbus-connection - [~nimbus-sym (*STORM-CONF* NIMBUS-HOST) (*STORM-CONF* NIMBUS-THRIFT-PORT)] - ~@body)) + `(let [context# (ReqContext/context) + user# (if (.principal context#) (.getName (.principal context#)))] +(thrift/with-nimbus-connection-as-user + [~nimbus-sym (*STORM-CONF* NIMBUS-HOST) (*STORM-CONF* NIMBUS-THRIFT-PORT) user#] + ~@body))) (defn assert-authorized-user ([servlet-request op] (assert-authorized-user servlet-request op nil)) ([servlet-request op topology-conf] - (if http-creds-handler (.populateContext http-creds-handler (ReqContext/context) servlet-request)) - (if *UI-ACL-HANDLER* - (let [context (ReqContext/context)] - (if-not (.permit *UI-ACL-HANDLER* context op topology-conf) - (let [principal (.principal context) - user (if principal (.getName principal) unknown)] - (throw (AuthorizationException. - (str UI request ' op ' for ' - user ' user is not authorized) +(let [context (ReqContext/context)] + (if http-creds-handler (.populateContext http-creds-handler context servlet-request)) + + (if (.isImpersonating context) +(if *UI-IMPERSONATION-HANDLER* +(if-not (.permit *UI-IMPERSONATION-HANDLER* context op topology-conf) + (let [principal (.principal context) +real-principal (.realPrincipal context) +user (if principal (.getName principal) unknown) +real-user (if real-principal (.getName real-principal) unknown) +remote-address (.remoteAddress context)] +(throw (AuthorizationException. + (str user ' real-user ' is not authorized to impersonate user ' user ' from host ' remote-address '. Please + see SECURITY.MD to learn how to configure impersonation ACL.) + (log-warn principal (.realPrincipal context) is trying to impersonate (.principal context) but --- End diff -- Again do we want to fail open or fail closed? secure Impersonation in storm - Key: STORM-446 URL: https://issues.apache.org/jira/browse/STORM-446 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Parth Brahmbhatt Labels: Security Storm security adds features of authenticating with kerberos and than uses that principal and TGT as way to authorize user operations, topology operation. Currently Storm UI user needs to be part of nimbus.admins to get details on user submitted topologies. Ideally storm ui needs to take authenticated user principal to submit requests to nimbus which will than authorize the user rather than storm UI user. This feature will also benefit superusers to impersonate other users to submit topologies in a secured way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-446: Allow superusers to impersonate oth...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/448#discussion_r25613289
--- Diff: storm-core/src/clj/backtype/storm/daemon/nimbus.clj ---
@@ -778,9 +779,22 @@
(defn check-authorization!
([nimbus storm-name storm-conf operation context]
(let [aclHandler (:authorization-handler nimbus)
+ impersonation-authorizer (:impersonation-authorization-handler
nimbus)
ctx (or context (ReqContext/context))
check-conf (if storm-conf storm-conf (if storm-name
{TOPOLOGY-NAME storm-name}))]
(log-message [req (.requestID ctx) ] Access from:
(.remoteAddress ctx) principal: (.principal ctx) op: operation)
+
+ (if (.isImpersonating ctx)
+ (do
+ (log-warn principal: (.realPrincipal ctx) is trying to
impersonate principal: (.principal ctx))
+ (if impersonation-authorizer
+ (if-not (.permit impersonation-authorizer ctx operation
check-conf)
+ (throw (AuthorizationException. (str principal
(.realPrincipal ctx) is not authorized to impersonate
+principal (.principal ctx) from host
(.remoteAddress ctx) Please see SECURITY.MD to learn
+how to configure impersonation acls.
+ (log-warn impersonation attempt but
NIMBUS-IMPERSONATION-AUTHORIZER has no authorizer configured. potential
--- End diff --
Do we want to fail open or fail closed in this case. For me if someone
tries to do impersonation and we have not configured anything for this I would
rather fail closed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/314 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-554) the get-task-object function in task.clj,the type of first param topology should be ^StormTopology not ^TopologyContext
[ https://issues.apache.org/jira/browse/STORM-554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343391#comment-14343391 ] ASF GitHub Bot commented on STORM-554: -- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/433#issuecomment-76752333 @BuDongDong Would you add [STORM-554] to the title of this pull request, and close #312 ? the get-task-object function in task.clj,the type of first param topology should be ^StormTopology not ^TopologyContext --- Key: STORM-554 URL: https://issues.apache.org/jira/browse/STORM-554 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.3-rc2 Reporter: zhangjinlong Assignee: zhangjinlong Fix For: 0.9.3-rc2 the get-task-object function in task.clj,the type of first param topology should be ^StormTopology not ^TopologyContext -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-557] Created docs directory and added i...
Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/314#issuecomment-76749927 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-446) secure Impersonation in storm
[ https://issues.apache.org/jira/browse/STORM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343371#comment-14343371 ] ASF GitHub Bot commented on STORM-446: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/448#issuecomment-76749638 For the most part it looks great. Just a little curious if we really want to fail open when no impersonation authorizer is configured? I am +1 either way. secure Impersonation in storm - Key: STORM-446 URL: https://issues.apache.org/jira/browse/STORM-446 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Parth Brahmbhatt Labels: Security Storm security adds features of authenticating with kerberos and than uses that principal and TGT as way to authorize user operations, topology operation. Currently Storm UI user needs to be part of nimbus.admins to get details on user submitted topologies. Ideally storm ui needs to take authenticated user principal to submit requests to nimbus which will than authorize the user rather than storm UI user. This feature will also benefit superusers to impersonate other users to submit topologies in a secured way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[
https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343386#comment-14343386
]
ASF GitHub Bot commented on STORM-682:
--
Github user kishorvpatil commented on the pull request:
https://github.com/apache/storm/pull/437#issuecomment-76751433
Looks good. +1
Supervisor local worker state corrupted and failing to start.
-
Key: STORM-682
URL: https://issues.apache.org/jira/browse/STORM-682
Project: Apache Storm
Issue Type: Bug
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt
If supervisor's cleanup of a worker fails to delete some heartbeat files the
local state of the supervisors get corrupted.The only way to recover the
supervisor from this state is to delete the local state folder where
supervisor stores all worker information.This fix can get very cumbersome if
it happens on multiple worker nodes.
The root cause of the issue is the order in which worker heartbeat versioned
store files are created vs the deletion order of those files. LocalState.put
first creates a data file X and then marks a success by creating a file
X.version. During get it first checks for all *.version files , tries to
find the largest value of X and then issues a read against X. See the below
pseudo code
{code:java}
start_supervisor() {
workerIds = `ls local-state/workers`
for each workerId in workerIds
versions = `ls local-state/workers/workerId/heartbeats/*.version`
latest_version = max(versions)
read local-state/workers/workerId/heartbeats/latest_version [Note there
is no .version extension]
}
{code}
During cleanup it first tries to delete file X and then X.version. If X gets
deleted but X.version fails to delete the supervisor fails to start with
FileNotFoundException in the code above.
We propose to change the deletion order so the .version files get deleted
before the data file and catch any IOException when reading worker heartbeats
to avoid supervisor failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (STORM-557) High Quality Images for presentations, etc.
[ https://issues.apache.org/jira/browse/STORM-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishor Patil resolved STORM-557. Resolution: Fixed High Quality Images for presentations, etc. --- Key: STORM-557 URL: https://issues.apache.org/jira/browse/STORM-557 Project: Apache Storm Issue Type: Documentation Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.10.0 Recently I created a couple of svg diagrams for a poster I was doing about secure storm. I thought it was a complete waste to not release them as open source, but I wasn't totally sure where we wanted to keep them. I can check them into git, because that would make it simple for others to find, and we could link to them from the markdown documentation. But I could also put them in the svn repo for the official site documentation. I'll throw up a very basic pull request and see what people think. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-446) secure Impersonation in storm
[ https://issues.apache.org/jira/browse/STORM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343491#comment-14343491 ] ASF GitHub Bot commented on STORM-446: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/448#issuecomment-76764354 We should fail close, however currently if the acl authorizer is no set we fail open. Not sure why did we make that choice but I did not want alternating behavior. My personal preference is to fail close on any potential security configuration mistake. secure Impersonation in storm - Key: STORM-446 URL: https://issues.apache.org/jira/browse/STORM-446 Project: Apache Storm Issue Type: Improvement Reporter: Sriharsha Chintalapani Assignee: Parth Brahmbhatt Labels: Security Storm security adds features of authenticating with kerberos and than uses that principal and TGT as way to authorize user operations, topology operation. Currently Storm UI user needs to be part of nimbus.admins to get details on user submitted topologies. Ideally storm ui needs to take authenticated user principal to submit requests to nimbus which will than authorize the user rather than storm UI user. This feature will also benefit superusers to impersonate other users to submit topologies in a secured way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-446: Allow superusers to impersonate oth...
Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/448#issuecomment-76764354 We should fail close, however currently if the acl authorizer is no set we fail open. Not sure why did we make that choice but I did not want alternating behavior. My personal preference is to fail close on any potential security configuration mistake. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Merge pull request #1 from apache/0.9.3-branch
Github user yaochitc closed the pull request at: https://github.com/apache/storm/pull/452 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
