We added in the symlinks copying what Hadoop does for their distributed cache,
and they too require the same set of permissions to make symlinks and Hadoop
work on windows. As such we didn't think it would be that big of an issue, but
we don't use windows here at Yahoo and it is not that simple to get access to a
windows box here so we didn't take the time to test out how painful it is.
Currently it is not just a nice to have part of the blobstore. If a user wants
to have access to a blob it shows up in the current working directory being
pointed to by a symlink. This also allows us to both share the blobs among
different workers running on the same box and atomically switch to an upgraded
version of the file. Additionally the new worker-artifacts feature uses
symlinks to keep files that a user wants to expose for download/debugging
separate from temporary files that will go away when the worker is shot.
Worker-artifacts we could rework to not use symlinks, but then we could not
configure it to put the worker logs under the logs directory by default they
would have to be hard coded to the worker directory and then moved when the
worker finishes. At Yahoo we use the blob-store with dynamic update and
compressed archives all the time and cannot remove it.
Hard links are not an option because you cannot point to directories, which is
a requirement for exposing uncompressed archives to end users. Perhaps what we
could do is change worker-artifacts to not use symlinks, and then have an
option to disable symlinks creation but at the same time also create a file
that users/workers could read instead of the symlink to know where a blob is
downloaded (a symlink without actually being a symlink). We would have to
expose a library that would make it simple for end users to know how to get to
their blob data, but that should not be too difficult.
- Bobby
On Tuesday, March 15, 2016 8:36 AM, Jungtaek Lim <[email protected]> wrote:
Hi devs,
I'm experimenting current 1.x-branch on Windows 8 / Oracle Java 1.8.0_73
and there seems be some issues regarding BlobStore which makes *supervisor
not functioning*.
(I believe this relates to failures of BlobStore UT on Windows.)
I fixed one issue regarding moving non-empty directory (will submit PR) and
went ahead, but met symbolic link issue immediately.
Try some googling, but unfortunately making symlink seems to be complicated
in Windows.
- If account's group is administrator, account should make symlink with
running as Administrator.
- I ran supervisor by account which group is administrator, and it
throws FileSystemException: A required privilege is not held by the client.
- It runs fine when I run supervisor from *elevated command prompt*.
- If account's group is not administrator, user should add that account to
the list of accounts/groups of "Create symbolic links". Details:
http://superuser.com/a/105381
I just feel both of two options seems not ideal. According to options,
users should do something to make Storm run well with Windows.
What would be better way to handle symbolic link?
We can remove creating symlink if not mandatory, but I'm curious it's
convenient for *Nix users.
Please share your opinions.
Thanks!
Jungtaek Lim (HeartSaVioR)
