Re: Starting sparkthrift server
we are running this right now as root user and the folder /tmp/spark-events was manually created and the Job has access to this folder On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee denny.g@gmail.com wrote: It appears that you are running the thrift-server using the spark-events account but the /tmp/spark-events folder doesn't exist or the user running thrift-server does not have access to it. Have you been able to run Hive using the spark-events user so that way the /tmp/spark-events folder has been created. If you need to reassign the scratch dir / log dir to another folder (instead of /tmp/spark-events), you could use --hiveconf to assign those to another folder. On Mon, Mar 23, 2015 at 8:39 AM Neil Dev neilk...@gmail.com wrote: Hi, I am having issues starting spark-thriftserver. I'm running spark 1.3.o with Hadoop 2.4.0. I would like to be able to change its port too so, I can hive hive-thriftserver as well as spark-thriftserver running at the same time. Starting sparkthrift server:- sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077 --executor-memory 2G Error:- I created the folder manually but still getting the following error Exception in thread main java.lang.IllegalArgumentException: Log directory /tmp/spark-events does not exist. I am getting the following error 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error: org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address0.0.0.0/0.0.0.0:1. at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:93) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:79) at org.apache.hive.service.auth.HiveAuthFactory.getServerSocket( HiveAuthFactory.java:236) at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService. run(ThriftBinaryCLIService.java:69) at java.lang.Thread.run(Thread.java:745) Thanks Neil
Re: Starting sparkthrift server
When you say the job has access, do you mean that when you run spark-submit or spark-shell (for example), it is able to write to the /tmp/spark-events folder? On Mon, Mar 23, 2015 at 1:02 PM Neil Dev neilk...@gmail.com wrote: we are running this right now as root user and the folder /tmp/spark-events was manually created and the Job has access to this folder On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee denny.g@gmail.com wrote: It appears that you are running the thrift-server using the spark-events account but the /tmp/spark-events folder doesn't exist or the user running thrift-server does not have access to it. Have you been able to run Hive using the spark-events user so that way the /tmp/spark-events folder has been created. If you need to reassign the scratch dir / log dir to another folder (instead of /tmp/spark-events), you could use --hiveconf to assign those to another folder. On Mon, Mar 23, 2015 at 8:39 AM Neil Dev neilk...@gmail.com wrote: Hi, I am having issues starting spark-thriftserver. I'm running spark 1.3.o with Hadoop 2.4.0. I would like to be able to change its port too so, I can hive hive-thriftserver as well as spark-thriftserver running at the same time. Starting sparkthrift server:- sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077 --executor-memory 2G Error:- I created the folder manually but still getting the following error Exception in thread main java.lang.IllegalArgumentException: Log directory /tmp/spark-events does not exist. I am getting the following error 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error: org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address0.0.0.0/0.0.0.0:1. at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:93) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:79) at org.apache.hive.service.auth.HiveAuthFactory.getServerSocket( HiveAuthFactory.java:236) at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService. run(ThriftBinaryCLIService.java:69) at java.lang.Thread.run(Thread.java:745) Thanks Neil
Re: Starting sparkthrift server
When I start spark-shell (for example) it does not write to the /tmp/spark-events folder. It remains empty. I have even tried it after giving that folder rwx permission for user, group and others. Neil's colleague, Anu On Mon, Mar 23, 2015 at 4:50 PM, Denny Lee denny.g@gmail.com wrote: When you say the job has access, do you mean that when you run spark-submit or spark-shell (for example), it is able to write to the /tmp/spark-events folder? On Mon, Mar 23, 2015 at 1:02 PM Neil Dev neilk...@gmail.com wrote: we are running this right now as root user and the folder /tmp/spark-events was manually created and the Job has access to this folder On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee denny.g@gmail.com wrote: It appears that you are running the thrift-server using the spark-events account but the /tmp/spark-events folder doesn't exist or the user running thrift-server does not have access to it. Have you been able to run Hive using the spark-events user so that way the /tmp/spark-events folder has been created. If you need to reassign the scratch dir / log dir to another folder (instead of /tmp/spark-events), you could use --hiveconf to assign those to another folder. On Mon, Mar 23, 2015 at 8:39 AM Neil Dev neilk...@gmail.com wrote: Hi, I am having issues starting spark-thriftserver. I'm running spark 1.3.o with Hadoop 2.4.0. I would like to be able to change its port too so, I can hive hive-thriftserver as well as spark-thriftserver running at the same time. Starting sparkthrift server:- sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077 --executor-memory 2G Error:- I created the folder manually but still getting the following error Exception in thread main java.lang.IllegalArgumentException: Log directory /tmp/spark-events does not exist. I am getting the following error 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error: org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address0.0.0.0/0.0.0.0:1. at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:93) at org.apache.thrift.transport.TServerSocket.init(TServerSocket.java:79) at org.apache.hive.service.auth.HiveAuthFactory.getServerSocket( HiveAuthFactory.java:236) at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService. run(ThriftBinaryCLIService.java:69) at java.lang.Thread.run(Thread.java:745) Thanks Neil
