[
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987718#comment-14987718
]
ASF GitHub Bot commented on STORM-876:
--------------------------------------
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/storm/pull/845#discussion_r43781137
--- Diff: storm-core/src/clj/backtype/storm/cluster.clj ---
@@ -593,18 +612,22 @@
(let [thrift-assignment (thriftify-assignment info)]
(set-data cluster-state (assignment-path storm-id)
(Utils/serialize thrift-assignment) acls)))
- (setup-code-distributor!
- [this storm-id nimbusInfo]
- (let [path (str (code-distributor-path storm-id) "/"
(.toHostPortString nimbusInfo))]
- (mkdirs cluster-state (code-distributor-path storm-id) acls)
- ;we delete the node first to ensure the node gets created as part
of this session only.
- (delete-node cluster-state path)
- (set-ephemeral-node cluster-state path nil acls)))
+ (setup-blobstore!
+ [this key nimbusInfo versionInfo]
+ (let [path (str (blobstore-path key) "/" (.toHostPortString
nimbusInfo) "-" versionInfo)]
+ (mkdirs cluster-state (blobstore-path key) acls)
+ ;we delete the node first to ensure the node gets created as
part of this session only.
+ (delete-node-blobstore cluster-state (str (blobstore-path key))
(.toHostPortString nimbusInfo))
+ (set-ephemeral-node cluster-state path nil acls)))
+
+ (remove-blobstore-key!
+ [this blob-key]
+ (log-debug "removing key" blob-key)
+ (delete-node cluster-state (blobstore-path blob-key)))
(remove-storm!
[this storm-id]
(delete-node cluster-state (assignment-path storm-id))
- (delete-node cluster-state (code-distributor-path storm-id))
--- End diff --
(schedule-recurring (:timer nimbus)
0
(conf NIMBUS-MONITOR-FREQ-SECS)
(fn []
(when-not (conf NIMBUS-DO-NOT-REASSIGN)
(locking (:submit-lock nimbus)
(mk-assignments nimbus)))
(do-cleanup nimbus) --> The clean up code takes
care of deleting the blobs
))
> Dist Cache: Basic Functionality
> -------------------------------
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: DISTCACHE.md, DistributedCacheDesignDocument.pdf
>
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and
> downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar
> should be written into the blob store instead of residing locally. We need a
> default implementation of the blob store that does essentially what nimbus
> currently does and does not need anything extra. But having an HDFS backend
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and
> provide a working directory for the worker process with symlinks to the
> blobs. It should also allow the blobs to be updated and switch the symlink
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process
> of getting it ready to push back to open source shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
