[
https://issues.apache.org/jira/browse/WHIRR-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Amos updated WHIRR-330:
----------------------------
Component/s: core
Description:
I created a hadoop-byon.properties file in recipes similar to
recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for
my machines, then I ran:
{noformat}
bin/whirr launch-cluster --config recipes/hadoop-byon.properties
{noformat}
I see the following error:
{noformat}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
at
org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
at
org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
at
org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:68)
at org.apache.whirr.cli.Main.main(Main.java:106)
{noformat}
The code looks like this:
{code:java}
Hardware hardware = Iterables.getFirst(taskTrackers, null)
.getNodeMetadata().getHardware();
int coresPerNode = 0;
for (Processor processor : hardware.getProcessors()) {
coresPerNode += processor.getCores();
}
{code}
The problem is that ByonClusterAction.java passes a null NodeMetadata when it
creates a new Instance:
{code:java}
public Instance apply(NodeMetadata node) {
String publicIp = Iterables.get(node.getPublicAddresses(), 0);
return new Instance(credentials, roles, publicIp, publicIp, node.getId(),
null);
}
{code}
I changed null to node in the above code and it only helps a little because
then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109
because the NodeMetadata has a null Hardware attribute. It seems like one of
two things could happen:
(1) recipes/nodes-byon.yaml could be enhanced to allow specification of
hardware, including processors and cores per processor (and the first
NullPointerException at line 106 would still need to be fixed)
(2) The machines could be queried at runtime using /proc/cpuinfo, but that
would probably need to happen in jclouds.
was:
I created a hadoop-byon.properties file in recipes similar to
recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml for
my machines, then I ran:
{noformat}
bin/whirr launch-cluster --config recipes/hadoop-byon.properties
{noformat}
I see the following error:
{noformat}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
at
org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
at
org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
at
org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
at
org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
at
org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
at org.apache.whirr.cli.Main.run(Main.java:68)
at org.apache.whirr.cli.Main.main(Main.java:106)
{noformat}
The code looks like this:
{code:java}
Hardware hardware = Iterables.getFirst(taskTrackers, null)
.getNodeMetadata().getHardware();
int coresPerNode = 0;
for (Processor processor : hardware.getProcessors()) {
coresPerNode += processor.getCores();
}
{code}
The problem is that ByonClusterAction.java passes a null NodeMetadata when it
creates a new Instance:
{code:java}
public Instance apply(NodeMetadata node) {
String publicIp = Iterables.get(node.getPublicAddresses(), 0);
return new Instance(credentials, roles, publicIp, publicIp,
node.getId(), null);
}
{code}
I changed null to node in the above code and it only helps a little because
then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109
because the NodeMetadata has a null Hardware attribute. It seems like one of
two things could happen:
(1) recipes/nodes-byon.yaml could be enhanced to allow specification of
hardware, including processors and cores per processor (and the first
NullPointerException at line 106 would still need to be fixed)
(2) The machines could be queried at runtime using /proc/cpuinfo, but that
would probably need to happen in jclouds.
> BYON doesn't work with HadoopConfigurationBuilder
> -------------------------------------------------
>
> Key: WHIRR-330
> URL: https://issues.apache.org/jira/browse/WHIRR-330
> Project: Whirr
> Issue Type: Bug
> Components: core, service/hadoop
> Affects Versions: 0.5.0
> Environment: CentOS 5.5
> Reporter: John Amos
>
> I created a hadoop-byon.properties file in recipes similar to
> recipes/zookeeper-byon.properties, and I customized recipes/nodes-byon.yaml
> for my machines, then I ran:
> {noformat}
> bin/whirr launch-cluster --config recipes/hadoop-byon.properties
> {noformat}
> I see the following error:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduceConfiguration(HadoopConfigurationBuilder.java:106)
> at
> org.apache.whirr.service.hadoop.HadoopConfigurationBuilder.buildMapReduce(HadoopConfigurationBuilder.java:71)
> at
> org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.beforeConfigure(HadoopNameNodeClusterActionHandler.java:80)
> at
> org.apache.whirr.service.ClusterActionHandlerSupport.beforeAction(ClusterActionHandlerSupport.java:48)
> at
> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:75)
> at
> org.apache.whirr.ByonClusterController.launchCluster(ByonClusterController.java:54)
> at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:61)
> at org.apache.whirr.cli.Main.run(Main.java:68)
> at org.apache.whirr.cli.Main.main(Main.java:106)
> {noformat}
> The code looks like this:
> {code:java}
> Hardware hardware = Iterables.getFirst(taskTrackers, null)
> .getNodeMetadata().getHardware();
> int coresPerNode = 0;
> for (Processor processor : hardware.getProcessors()) {
> coresPerNode += processor.getCores();
> }
> {code}
> The problem is that ByonClusterAction.java passes a null NodeMetadata when it
> creates a new Instance:
> {code:java}
> public Instance apply(NodeMetadata node) {
> String publicIp = Iterables.get(node.getPublicAddresses(), 0);
> return new Instance(credentials, roles, publicIp, publicIp, node.getId(),
> null);
> }
> {code}
> I changed null to node in the above code and it only helps a little because
> then a NullPointerException is thrown at HadoopConfigurationBuilder.java:109
> because the NodeMetadata has a null Hardware attribute. It seems like one of
> two things could happen:
> (1) recipes/nodes-byon.yaml could be enhanced to allow specification of
> hardware, including processors and cores per processor (and the first
> NullPointerException at line 106 would still need to be fixed)
> (2) The machines could be queried at runtime using /proc/cpuinfo, but that
> would probably need to happen in jclouds.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
