[
https://issues.apache.org/jira/browse/STORM-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated STORM-876:
--------------------------------------
Description:
Basic functionality for the Dist Cache feature.
As part of this a new API should be added to support uploading and downloading
dist cache items. storm-core.ser, storm-conf.ser and storm.jar should be
written into the bolb store instead of residing locally. We need a default
implementation of the blob store that does essentially what nimbus currently
does and does not need anything extra. But having an HDFS backend too would be
great for scalability and HA.
The supervisor should provide a way to download and manage these blobs and
provide a working directory for the worker process with symlinks to the blobs.
It should also allow the blobs to be updated and switch the symlink atomically
to point to the new blob once it is downloaded.
All of this is already done by code internal to Yahoo! we are in the process of
getting it ready to push back to open source shortly.
was:
Basic functionality for the Dist Cache feature.
As part of this a new API should be added to support uploading and downloading
dist cache items. storm-core.ser, storm-conf.ser and storm.jar should be
written into the bolb store instead of residing locally. We need a default
implementation of the blob store that does essentially what nimbus currently
does and does not need anything extra. But having an HDFS backend too would be
great for scalability and HA.
All of this is already done by code internal to Yahoo! we are in the process of
getting it ready to push back to open source shortly.
> Dist Cache: Basic Functionality
> -------------------------------
>
> Key: STORM-876
> URL: https://issues.apache.org/jira/browse/STORM-876
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
>
> Basic functionality for the Dist Cache feature.
> As part of this a new API should be added to support uploading and
> downloading dist cache items. storm-core.ser, storm-conf.ser and storm.jar
> should be written into the bolb store instead of residing locally. We need a
> default implementation of the blob store that does essentially what nimbus
> currently does and does not need anything extra. But having an HDFS backend
> too would be great for scalability and HA.
> The supervisor should provide a way to download and manage these blobs and
> provide a working directory for the worker process with symlinks to the
> blobs. It should also allow the blobs to be updated and switch the symlink
> atomically to point to the new blob once it is downloaded.
> All of this is already done by code internal to Yahoo! we are in the process
> of getting it ready to push back to open source shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
