In that case, can you use the configurations to specify the folders? I'm wondering if this actually Hive in play here and somehow the /tmp/spark-events is being specified for the logs for Hive?
On Mon, Mar 23, 2015 at 2:00 PM Anubhav Agarwal <anubha...@gmail.com> wrote: > When I start spark-shell (for example) it does not write to the > /tmp/spark-events folder. It remains empty. I have even tried it after > giving that folder rwx permission for user, group and others. > > Neil's colleague, > Anu > > On Mon, Mar 23, 2015 at 4:50 PM, Denny Lee <denny.g....@gmail.com> wrote: > >> When you say the job has access, do you mean that when you run >> spark-submit >> or spark-shell (for example), it is able to write to the /tmp/spark-events >> folder? >> >> >> On Mon, Mar 23, 2015 at 1:02 PM Neil Dev <neilk...@gmail.com> wrote: >> >> > we are running this right now as root user and the folder >> /tmp/spark-events >> > was manually created and the Job has access to this folder >> > >> > On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <denny.g....@gmail.com> >> wrote: >> > >> >> It appears that you are running the thrift-server using the >> spark-events >> >> account but the /tmp/spark-events folder doesn't exist or the user >> running >> >> thrift-server does not have access to it. Have you been able to run >> Hive >> >> using the spark-events user so that way the /tmp/spark-events folder >> has >> >> been created. If you need to reassign the scratch dir / log dir to >> another >> >> folder (instead of /tmp/spark-events), you could use --hiveconf to >> assign >> >> those to another folder. >> >> >> >> >> >> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <neilk...@gmail.com> wrote: >> >> >> >>> Hi, >> >>> >> >>> I am having issues starting spark-thriftserver. I'm running spark >> 1.3.o >> >>> with Hadoop 2.4.0. I would like to be able to change its port too so, >> I >> >>> can >> >>> hive hive-thriftserver as well as spark-thriftserver running at the >> same >> >>> time. >> >>> >> >>> Starting sparkthrift server:- >> >>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077 >> >>> --executor-memory 2G >> >>> >> >>> Error:- >> >>> I created the folder manually but still getting the following >> error---- >> >>> Exception in thread "main" java.lang.IllegalArgumentException: Log >> >>> directory /tmp/spark-events does not exist. >> >>> >> >>> >> >>> I am getting the following error >> >>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error: >> >>> org.apache.thrift.transport.TTransportException: Could not create >> >>> ServerSocket on address0.0.0.0/0.0.0.0:10000. >> >>> at >> >>> >> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93) >> >>> at >> >>> >> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79) >> >>> at >> >>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket( >> >>> HiveAuthFactory.java:236) >> >>> at >> >>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService. >> >>> run(ThriftBinaryCLIService.java:69) >> >>> at java.lang.Thread.run(Thread.java:745) >> >>> >> >>> Thanks >> >>> Neil >> >>> >> >> >> > >> > >
