I agree that running storm on windows is not ideal in its current incarnation.
We decided to use symlinks because Hadoop uses symlinks similarly, and thought
that it would not be too much of a burden for people using windows. It turns
out we were wrong. - Bobby
On Friday, August 12, 2016 10:21 AM, "[email protected]"
<[email protected]> wrote:
Bobby,
Thanks for the response.
I appreciate that using symlinks for things like this makes some things far
easier, and on operating systems other than Windows this is generally not a
problem (honestly, the biggest surprise for me in this whole thing was that
Windows locks down symlinks so much). I'll see about raising a Jira at some
point for the worker artifacts aspect, but if additional functionality it going
to require this down the line, then things could become more troublesome in the
future. It makes using Storm in a secured environment (running Windows!?!)
difficult, as justifying changes to group policies requires jumping through
numerous hoops.
Thanks
Paul
-----Original Message-----
From: Bobby Evans [mailto:[email protected]]
Sent: 12 August 2016 15:27
To: [email protected]
Subject: Re: Unable to run Storm 1.0.1 on Windows
Paul,
There currently is no way to disable the symlink functionality out of the box
in 1.x. It is primarily used in two different places in storm, in pointing the
worker to the workers-artifacts directory, and also in using blobs from the
distributed cache. We expect more and more features to be built on top of
these features so having symlinks be optional is not necessarily all that
trivial. For the extra blobs it would require you to not use that feature at
all. There are some key parts of the functionality that do not work without
symlinks. For the worker artifacts we could provide an alternative way to tell
the worker where this is, in addition to the symlink. That would not be too
bad to do.
If you want to you can file a JIRA for this that would be the best way, but I
honestly don't know when someone would get around to working on it. If you are
getting commercial support from someone you might want to contact them too. -
Bobby
On Friday, August 12, 2016 4:56 AM, "[email protected]"
<[email protected]> wrote:
Hi all,
(Following this up on the dev list, as I didn't receive any response on the
user list.)
We hoped to upgrade our system to Storm 1.0 (or above), however, due to
security hardening on our systems (and client systems outside of our control),
the documented methods to allow the blob store functionality to work (by making
a group policy change to enable symlinks) on Windows are not possible for us.
Is there any way to avoid the need for newer versions of Storm to create
symlinks? Is there any way to disable this functionality if we don't explicitly
require it? I couldn't find anything when looking through documentation other
than the options run change group policy or run the supervisor as an
administrator user.
Regards
Paul Milliken
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: 20 July 2016 07:59
To: [email protected]
Subject: RE: Unable to run Storm 1.0.1 on Windows
Hi,
Just a quick summary/follow up in case the wall of text sent previously put
people off.
Is there any way to disable / avoid the need to create symlinks in newer
versions of Storm so that it can be used in a more locked down Windows
environment?
Thanks
Paul Milliken
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: 18 July 2016 15:23
To: [email protected]
Subject: Unable to run Storm 1.0.1 on Windows
Hello,
We are using Storm as a core component of a large-scale document
processing/analysis system. We have been developing this for a while now,
starting with Storm 0.8 (I think). We are currently using Storm 0.10.0 and were
hoping to upgrade to 1.0.1 (as the release notes indicate that some of the
changes will potentially improve performance).
Unfortunately, the new version does not run in our development environments as
it now requires the ability to create symbolic links. The documentation at
https://github.com/apache/storm/blob/master/docs/windows-users-guide.md
indicates that this can be made to work by either running as an administrator
or by changing a group policy setting. However, neither of these approaches is
possible in our case, since we are working in a relatively locked-down
environment. Running as an administrator is definitely not an appropriate
workaround - not all our developers have administrator access (and even if they
did, I would be very hesitant to advise them to run things as administrator).
Our development process also means that we regularly stop, wipe out and restart
the cluster, so we can't just run the supervisor once as a different user with
the relevant permissions. Our intention is that things work out of the box (ie,
unzip and run a script to get a working system) without needing to set up
certain processes to be run separately.
For added complication, the product that my team produces is then taken by
consulting teams and potentially customized for each client. This work often
happens on client sites, where being able to run as an administrator or change
group policies is even less likely. In this scenario, simply being able to
unpack a complete development environment with minimal requirements on the
system is critical.
All that said, is there any other way of getting newer versions of Storm to
work on Windows without needing to either run as an administrator or make group
policy changes? Since we started our development on earlier versions, we are
not making any direct use of the functionality that this supports (we don't
need to share filesystem resources between workers, since all our shared state
is being kept in a database). Is there any way to disable the symlink creation,
or have it fall back to just copying data instead?
Thanks in advance,
Paul Milliken
Please consider the environment before printing this email. This message should
be regarded as confidential. If you have received this email in error please
notify the sender and destroy it immediately. Statements of intent shall only
become binding when confirmed in hard copy by an authorised signatory. The
contents of this email may relate to dealings with other companies under the
control of BAE Systems Applied Intelligence Limited, details of which can be
found at http://www.baesystems.com/Businesses/index.htm.
