I tested that: I set:
_JAVA_OPTIONS=-Djava.net.preferIPv4Stack=true SPARK_LOCAL_IP=10.0.100.120 I still have the warning in the log: 16/07/10 14:10:13 WARN Utils: Your hostname, micha resolves to a loopback address: 127.0.1.1; using 10.0.100.120 instead (on interface eno1) 16/07/10 14:10:13 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address and still connection refused... but no luck > On Jul 10, 2016, at 1:26 PM, Jean Georges Perrin <j...@jgp.net> wrote: > > Hi, > > So far I have been using Spark "embedded" in my app. Now, I'd like to run it > on a dedicated server. > > I am that far: > - fresh ubuntu 16, server name is mocha / ip 10.0.100.120, installed scala > 2.10, installed Spark 1.6.2, recompiled > - Pi test works > - UI on port 8080 works > > Log says: > Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp > /opt/apache-spark-1.6.2/conf/:/opt/apache-spark-1.6.2/assembly/target/scala-2.10/spark-assembly-1.6.2-hadoop2.2.0.jar:/opt/apache-spark-1.6.2/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/opt/apache-spark-1.6.2/lib_managed/jars/datanucleus-core-3.2.10.jar:/opt/apache-spark-1.6.2/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar > -Xms1g -Xmx1g org.apache.spark.deploy.master.Master --ip micha --port 7077 > --webui-port 8080 > ======================================== > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 16/07/10 13:03:55 INFO Master: Registered signal handlers for [TERM, HUP, INT] > 16/07/10 13:03:55 WARN Utils: Your hostname, micha resolves to a loopback > address: 127.0.1.1; using 10.0.100.120 instead (on interface eno1) > 16/07/10 13:03:55 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to > another address > 16/07/10 13:03:55 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 16/07/10 13:03:55 INFO SecurityManager: Changing view acls to: root > 16/07/10 13:03:55 INFO SecurityManager: Changing modify acls to: root > 16/07/10 13:03:55 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(root); users > with modify permissions: Set(root) > 16/07/10 13:03:56 INFO Utils: Successfully started service 'sparkMaster' on > port 7077. > 16/07/10 13:03:56 INFO Master: Starting Spark master at spark://micha:7077 > <spark://micha:7077> > 16/07/10 13:03:56 INFO Master: Running Spark version 1.6.2 > 16/07/10 13:03:56 INFO Server: jetty-8.y.z-SNAPSHOT > 16/07/10 13:03:56 INFO AbstractConnector: Started > SelectChannelConnector@0.0.0.0 <mailto:SelectChannelConnector@0.0.0.0>:8080 > 16/07/10 13:03:56 INFO Utils: Successfully started service 'MasterUI' on port > 8080. > 16/07/10 13:03:56 INFO MasterWebUI: Started MasterWebUI at > http://10.0.100.120:8080 <http://10.0.100.120:8080/> > 16/07/10 13:03:56 INFO Server: jetty-8.y.z-SNAPSHOT > 16/07/10 13:03:56 INFO AbstractConnector: Started > SelectChannelConnector@micha:6066 > 16/07/10 13:03:56 INFO Utils: Successfully started service on port 6066. > 16/07/10 13:03:56 INFO StandaloneRestServer: Started REST server for > submitting applications on port 6066 > 16/07/10 13:03:56 INFO Master: I have been elected leader! New state: ALIVE > > > In my app, i changed the config to: > SparkConf conf = new > SparkConf().setAppName("myapp").setMaster("spark://10.0.100.120:6066 > <spark://10.0.100.120:6066>"); > > > (also tried 7077) > > > On the client: > 16-07-10 13:22:58:300 INFO org.spark-project.jetty.server.AbstractConnector - > Started SelectChannelConnector@0.0.0.0 > <mailto:SelectChannelConnector@0.0.0.0>:4040 > 16-07-10 13:22:58:300 DEBUG > org.spark-project.jetty.util.component.AbstractLifeCycle - STARTED > SelectChannelConnector@0.0.0.0 <mailto:SelectChannelConnector@0.0.0.0>:4040 > 16-07-10 13:22:58:300 DEBUG > org.spark-project.jetty.util.component.AbstractLifeCycle - STARTED > org.spark-project.jetty.server.Server@3eb292cd > 16-07-10 13:22:58:301 INFO org.apache.spark.util.Utils - Successfully started > service 'SparkUI' on port 4040. > 16-07-10 13:22:58:306 INFO org.apache.spark.ui.SparkUI - Started SparkUI at > http://10.0.100.100:4040 <http://10.0.100.100:4040/> > 16-07-10 13:22:58:621 INFO > org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to > master spark://10.0.100.120:6066 <spark://10.0.100.120:6066>... > 16-07-10 13:22:58:648 DEBUG > org.apache.spark.network.client.TransportClientFactory - Creating new > connection to /10.0.100.120:6066 > 16-07-10 13:22:58:689 DEBUG io.netty.util.ResourceLeakDetector - > -Dio.netty.leakDetectionLevel: simple > 16-07-10 13:22:58:714 WARN > org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect > to master 10.0.100.120:6066 > java.io.IOException: Failed to connect to /10.0.100.120:6066 > at > org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216) > > and if I try to telnet: > > $ telnet 10.0.100.120 6066 > Trying 10.0.100.120... > telnet: connect to address 10.0.100.120: Connection refused > telnet: Unable to connect to remote host > > $ telnet 10.0.100.120 7077 > Trying 10.0.100.120... > telnet: connect to address 10.0.100.120: Connection refused > telnet: Unable to connect to remote host > > On the server, I checked with netstat: > jgp@micha:/opt/apache-spark$ netstat -a | grep 6066 > tcp6 0 0 micha.nc.rr.com <http://micha.nc.rr.com/>:6066 [::]:* > LISTEN > jgp@micha:/opt/apache-spark$ netstat -a | grep 7077 > tcp6 0 0 micha.nc.rr.com <http://micha.nc.rr.com/>:7077 [::]:* > LISTEN > > If I interpret this, it looks like it listens in IP v6 and not 4... > > Any clue would be very helpful. I do not think I am that far, but... > > Thanks > > > jg > > > >
