[
https://issues.apache.org/jira/browse/WHIRR-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993943#comment-12993943
]
Tibor Kiss edited comment on WHIRR-168 at 2/12/11 4:36 PM:
-----------------------------------------------------------
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add
property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties.
Then I changed a line
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server",
clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to
hadoop-site.xml, unfortunately when the createClientSideProperties() is to be
writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<<
FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0
sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at
org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at
org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
After digging into the how HadoopConfigurationBuilder works, I found that
currently it is called only from
{code}
org.apache.whirr.service.hadoop.HadoopDataNodeClusterActionHandler.beforeConfigure(ClusterActionEvent)
{code}
and is completely missing the a buildClient(String path, ClusterSpec
clusterSpec, Cluster cluster) method from HadoopConfigurationBuilder, we only
have buildCommon, buildHdfs, buildMapReduce.
Similar to HadoopConfigurationBuilderTest do we have to implement that
functionality for client. I am right?
was (Author: tibor.kiss):
Hi Tom.
With the current trunk where we have WHIRR-55 added, I made a trial to add
property hadoop-common.hadoop.socks.server to whirr-hadoop-default.properties.
Then I changed a line
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
into
{code}
config.setProperty("hadoop.socks.server",
clusterSpec.getConfiguration().getString("hadoop.socks.server"));
{code}
I see in the starting nodes that the new default value is added to
hadoop-site.xml, unfortunately when the createClientSideProperties() is to be
writing the hadoop-proxy.sh file, the integration test fails with
{code}
-------------------------------------------------------------------------------
Test set: org.apache.whirr.service.hadoop.integration.HadoopServiceTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 549.834 sec <<<
FAILURE!
org.apache.whirr.service.hadoop.integration.HadoopServiceTest Time elapsed: 0
sec <<< ERROR!
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.createClientSideProperties(HadoopNameNodeClusterActionHandler.java:153)
at
org.apache.whirr.service.hadoop.HadoopNameNodeClusterActionHandler.afterConfigure(HadoopNameNodeClusterActionHandler.java:141)
at
org.apache.whirr.service.ClusterActionHandlerSupport.afterAction(ClusterActionHandlerSupport.java:48)
at
org.apache.whirr.cluster.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:87)
at org.apache.whirr.service.Service.launchCluster(Service.java:79)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.startup(HadoopServiceController.java:81)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceController.ensureClusterRunning(HadoopServiceController.java:66)
at
org.apache.whirr.service.hadoop.integration.HadoopServiceTest.setUp(HadoopServiceTest.java:56)
{code}
Can you explain me, why the mentioned
clusterSpec.getConfiguration().getString("hadoop.socks.server") returns null?
I was expecting that I can query that parameter from clusterSpec.
> Add a new optional c parameter for being able to configure the port of socks
> connection.
> ----------------------------------------------------------------------------------------
>
> Key: WHIRR-168
> URL: https://issues.apache.org/jira/browse/WHIRR-168
> Project: Whirr
> Issue Type: New Feature
> Components: core, service/hadoop
> Environment: ec2
> Reporter: Tibor Kiss
> Assignee: Tibor Kiss
> Priority: Minor
> Attachments: local-socks-proxy-address.patch
>
>
> We have a generated .whirr/<hadoop-cluster-name>/hadoop-proxy.sh which
> contains a hard coded port value, the 6666.
> In order to be able to start multiple clusters from the same console I needed
> a simple mechanism to be able to parametrize this port number.
> Therefore I made a patch which adds the possibility to set this
> 'whirr.local-socks-proxy-address' to something like
> whirr.local-socks-proxy-address=localhost:6666
> Instead of configuring the port, we are able to configure the address which
> contains the port.
> (also for the sourcecode, it looks much better to not have such a hardcoded
> value.)
> In order to run multiple clusters you only need to override this paramter
> knowing that the default value is localhost:6666
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
