Re: Starting Thriftserver via hostname on Spark 1.1 RC4?
Ahh got it - I knew I was missing something - appreciate the clarification! :) On September 4, 2014 at 10:27:44, Cheng Lian (lian.cs@gmail.com) wrote: You may configure listening host and port in the same way as HiveServer2 of Hive, namely: via environment variables HIVE_SERVER2_THRIFT_BIND_HOST HIVE_SERVER2_THRIFT_PORT via system properties hive.server2.thrift.bind.host hive.server2.thrift.port For the latter, you may pass the properties via command line: $ ./sbin/start-thriftserver2 --hiveconf hive.server2.thrift.bind.host --hiveconf hive.server2.thrift.port This behavior is inherited from Hive since Spark SQL Thrift server is a variant of HiveServer2. On Wed, Sep 3, 2014 at 10:47 PM, Denny Lee wrote: When I start the thrift server (on Spark 1.1 RC4) via: ./sbin/start-thriftserver.sh --master spark://hostname:7077 --driver-class-path $CLASSPATH It appears that the thrift server is starting off of localhost as opposed to hostname. I have set the spark-env.sh to use the hostname, modified the /etc/hosts for the hostname, and it appears to work properly. But when I start the thrift server, connectivity can only be via localhost:1 as opposed to hostname:1. Any ideas on what configurations I may be setting incorrectly here? Thanks! Denny
Re: Starting Thriftserver via hostname on Spark 1.1 RC4?
You may configure listening host and port in the same way as HiveServer2 of Hive, namely: - via environment variables - HIVE_SERVER2_THRIFT_BIND_HOST - HIVE_SERVER2_THRIFT_PORT - via system properties - hive.server2.thrift.bind.host - hive.server2.thrift.port For the latter, you may pass the properties via command line: $ ./sbin/start-thriftserver2 --hiveconf hive.server2.thrift.bind.host --hiveconf hive.server2.thrift.port This behavior is inherited from Hive since Spark SQL Thrift server is a variant of HiveServer2. On Wed, Sep 3, 2014 at 10:47 PM, Denny Lee wrote: > When I start the thrift server (on Spark 1.1 RC4) via: > ./sbin/start-thriftserver.sh --master spark://hostname:7077 > --driver-class-path $CLASSPATH > > It appears that the thrift server is starting off of localhost as opposed > to hostname. I have set the spark-env.sh to use the hostname, modified the > /etc/hosts for the hostname, and it appears to work properly. > > But when I start the thrift server, connectivity can only be via > localhost:1 as opposed to hostname:1. > > Any ideas on what configurations I may be setting incorrectly here? > > Thanks! > Denny > >
Re: Starting Thriftserver via hostname on Spark 1.1 RC4?
I want to add that there a regression when using pyspark to read data
from HDFS. its performance during map tasks has gone down approx 1 ->
0.5x. I have tested the 1.0.2 and the performance was fine, but the 1.1
release candidate has this issue. I tested by setting the following
properties to make sure it was not due to these.
set("spark.io.compression.codec","lzf").set("spark.shuffle.spill","false")
in conf object. Let me know if you need further information.
Regards,
Gurvinder
On 09/04/2014 07:47 AM, Denny Lee wrote:
> When I start the thrift server (on Spark 1.1 RC4) via:
> ./sbin/start-thriftserver.sh --master spark://hostname:7077
> --driver-class-path $CLASSPATH
>
> It appears that the thrift server is starting off of localhost as
> opposed to hostname. I have set the spark-env.sh to use the hostname,
> modified the /etc/hosts for the hostname, and it appears to work properly.
>
> But when I start the thrift server, connectivity can only be via
> localhost:1 as opposed to hostname:1.
>
> Any ideas on what configurations I may be setting incorrectly here?
>
> Thanks!
> Denny
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Starting Thriftserver via hostname on Spark 1.1 RC4?
When I start the thrift server (on Spark 1.1 RC4) via: ./sbin/start-thriftserver.sh --master spark://hostname:7077 --driver-class-path $CLASSPATH It appears that the thrift server is starting off of localhost as opposed to hostname. I have set the spark-env.sh to use the hostname, modified the /etc/hosts for the hostname, and it appears to work properly. But when I start the thrift server, connectivity can only be via localhost:1 as opposed to hostname:1. Any ideas on what configurations I may be setting incorrectly here? Thanks! Denny
