Hi Gary,
Thank you for your response. In the short run, I want to develop a twill
application on windows and then run it on a cluster. In the long run, I
probably will develop & test on windows using a local Hadoop, but first, I
would like to run hello world as a minimum working example.
Then I can build on this to do my own stuff, but before I do that, I want to go
through the examples to verify/fix that my setup works correctly, and that it
will run on the cluster correctly…
Which brings me to my actual question:
What do I have to do to get the hello world example to run on the real cluster?
I managed to get around a few issues. But it seems like the Hello World example
is optimized towards a local cluster installation.
So far I managed to fix (snippet at EOF):
- Connection to yarn fixed by setting the yarn resource manager
address to sth != 0.0.0.0:8032
- Connection to hdfs fixed by providing a defaultFS.
- Minimize the apps resource requirements, to make sure that yarn is
able to provide the needed resources.
Hello World will start, but not finish, nor print the log messages…
Open issues are therefore (from what I can see from here with my limited
experience):
- Fix log4j configurations so it stops complaining about “No
appenders” somehow.
Wrap-up: Hello World configuration for running on a real cluster seems to be
quite cumbersome and is still unkown. Any advice on this?
Best regards,
Hendrik
Twill configuration to get it running:
public static void main(String[] args) {
String zkConnectStr = "machine1:2181";
Configuration conf = new Configuration();
conf.set("yarn.resourcemanager.address", "machine1:8032");
conf.set("fs.defaultFS", "hdfs://machine1:8020");
YarnConfiguration yc = new YarnConfiguration(conf);
final TwillRunnerService twillRunner = new YarnTwillRunnerService(yc,
zkConnectStr);
twillRunner.startAndWait();
TwillPreparer twillPreparer = twillRunner.prepare(new HelloWorldRunnable(),
ResourceSpecification.Builder.with()
.setVirtualCores(1)
.setMemory(64, SizeUnit.MEGA)
.setInstances(1)
.build())
.addLogHandler(
new PrinterLogHandler(new PrintWriter(System.out, true)));
final TwillController controller = twillPreparer.start();
-----Ursprüngliche Nachricht-----
Von: Gary Helmling [mailto:[email protected]]
Gesendet: Freitag, 6. Juni 2014 18:39
An: [email protected]
Betreff: Re: Twill application development requirements for windows
Hi Hendrik,
Are you talking about developing a twill application on windows and then
running it on a cluster? That should be possible with just the Hadoop client
library jars (dependencies of twill).
Or are you talking about developing and testing locally on a pseudo-distributed
Hadoop cluster running on your windows box?
Please describe in a bit more detail what you want to do.
Gary
On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke <[email protected]>
wrote:
> Hi,
>
>
> is it possible to develop and start a twill application on windows
> without having to install hadoop binaries? Or a node needed on the local
> machine?
>
>
> Ty,
>
> Hendrik
