Hi, Still struggling to get Es + Hadoop smoothly working together.
Anyone facing this type of issue ? ERROR security.UserGroupInformation: PriviledgedActionException as:hue (auth:SIMPLE) cause:BeeswaxException 14/02/26 05:42:34 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=Driver.run> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=TimeToSubmit> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=compile> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=parse> > 14/02/26 05:42:34 INFO parse.ParseDriver: Parsing command: use default > 14/02/26 05:42:34 INFO parse.ParseDriver: Parse Completed > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=parse start=1393422154484 > end=1393422154485 duration=1> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=semanticAnalyze> > 14/02/26 05:42:34 INFO ql.Driver: Semantic Analysis Completed > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=semanticAnalyze > start=1393422154485 end=1393422154485 duration=0> > 14/02/26 05:42:34 INFO ql.Driver: Returning Hive schema: > Schema(fieldSchemas:null, properties:null) > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=compile > start=1393422154483 end=1393422154485 duration=2> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=Driver.execute> > 14/02/26 05:42:34 INFO ql.Driver: Starting command: use default > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=TimeToSubmit > start=1393422154483 end=1393422154486 duration=3> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=runTasks> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=task.DDL.Stage-0> > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=task.DDL.Stage-0 > start=1393422154486 end=1393422154493 duration=7> > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=runTasks > start=1393422154486 end=1393422154493 duration=7> > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=Driver.execute > start=1393422154485 end=1393422154493 duration=8> > OK > 14/02/26 05:42:34 INFO ql.Driver: OK > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=releaseLocks> > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=releaseLocks > start=1393422154493 end=1393422154493 duration=0> > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=Driver.run > start=1393422154483 end=1393422154493 duration=10> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=compile> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=parse> > 14/02/26 05:42:34 INFO parse.ParseDriver: Parsing command: INSERT OVERWRITE > TABLE eslogs SELECT s.time, s.ext, s.ip, s.req, s.res, s.agent FROM logs s > 14/02/26 05:42:34 INFO parse.ParseDriver: Parse Completed > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=parse start=1393422154494 > end=1393422154495 duration=1> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=semanticAnalyze> > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Starting Semantic Analysis > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Completed phase 1 of Semantic > Analysis > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Get metadata for source tables > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Get metadata for subqueries > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Get metadata for destination > tables > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Completed getting MetaData in > Semantic Analysis > 14/02/26 05:42:34 INFO ppd.OpProcFactory: Processing for FS(9) > 14/02/26 05:42:34 INFO ppd.OpProcFactory: Processing for SEL(8) > 14/02/26 05:42:34 INFO ppd.OpProcFactory: Processing for TS(7) > 14/02/26 05:42:34 INFO physical.MetadataOnlyOptimizer: Looking for table > scans where optimization is applicable > 14/02/26 05:42:34 INFO physical.MetadataOnlyOptimizer: Found 0 metadata only > table scans > 14/02/26 05:42:34 INFO parse.SemanticAnalyzer: Completed plan generation > 14/02/26 05:42:34 INFO ql.Driver: Semantic Analysis Completed > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=semanticAnalyze > start=1393422154495 end=1393422154549 duration=54> > 14/02/26 05:42:34 INFO ql.Driver: Returning Hive schema: > Schema(fieldSchemas:[FieldSchema(name:time, type:string, comment:null), > FieldSchema(name:ext, type:string, comment:null), FieldSchema(name:ip, > type:string, comment:null), FieldSchema(name:req, type:string, comment:null), > FieldSchema(name:res, type:int, comment:null), FieldSchema(name:agent, > type:string, comment:null)], properties:null) > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=compile > start=1393422154493 end=1393422154549 duration=56> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=Driver.execute> > 14/02/26 05:42:34 INFO ql.Driver: Starting command: INSERT OVERWRITE TABLE > eslogs SELECT s.time, s.ext, s.ip, s.req, s.res, s.agent FROM logs s > Total MapReduce jobs = 1 > 14/02/26 05:42:34 INFO ql.Driver: Total MapReduce jobs = 1 > 14/02/26 05:42:34 INFO ql.Driver: </PERFLOG method=TimeToSubmit > start=1393422154483 end=1393422154585 duration=102> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=runTasks> > 14/02/26 05:42:34 INFO ql.Driver: <PERFLOG method=task.MAPRED.Stage-0> > Launching Job 1 out of 1 > 14/02/26 05:42:34 INFO ql.Driver: Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > 14/02/26 05:42:34 INFO exec.Task: Number of reduce tasks is set to 0 since > there's no reduce operator > 14/02/26 05:42:34 INFO ql.Context: New scratch dir is > hdfs://sandbox.hortonworks.com:8020/tmp/hive-beeswax-hue/hive_2014-02-26_05-42-34_494_4288001753889524446-3 > 14/02/26 05:42:34 INFO Configuration.deprecation: > mapred.reduce.tasks.speculative.execution is deprecated. Instead, use > mapreduce.reduce.speculative > 14/02/26 05:42:34 INFO mr.ExecDriver: Using > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat > 14/02/26 05:42:34 INFO exec.Utilities: Processing alias s > 14/02/26 05:42:34 INFO exec.Utilities: Adding input file > hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/logs > 14/02/26 05:42:34 INFO exec.Utilities: Content Summary not cached for > hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/logs > 14/02/26 05:42:34 INFO ql.Context: New scratch dir is > hdfs://sandbox.hortonworks.com:8020/tmp/hive-beeswax-hue/hive_2014-02-26_05-42-34_494_4288001753889524446-3 > 14/02/26 05:42:34 INFO exec.Utilities: <PERFLOG method=serializePlan> > 14/02/26 05:42:34 INFO exec.Utilities: Serializing MapWork via kryo > 14/02/26 05:42:34 INFO exec.Utilities: </PERFLOG method=serializePlan > start=1393422154741 end=1393422154807 duration=66> > 14/02/26 05:42:34 INFO client.RMProxy: Connecting to ResourceManager at > sandbox.hortonworks.com/10.0.2.15:8050 > 14/02/26 05:42:34 INFO client.RMProxy: Connecting to ResourceManager at > sandbox.hortonworks.com/10.0.2.15:8050 > 14/02/26 05:42:34 WARN mapreduce.JobSubmitter: Hadoop command-line option > parsing not performed. Implement the Tool interface and execute your > application with ToolRunner to remedy this. > 14/02/26 05:42:35 INFO io.CombineHiveInputFormat: <PERFLOG method=getSplits> > 14/02/26 05:42:35 INFO io.CombineHiveInputFormat: CombineHiveInputSplit > creating pool for > hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/logs; using filter > path hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/logs > 14/02/26 05:42:35 INFO Configuration.deprecation: mapred.min.split.size is > deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize > 14/02/26 05:42:35 INFO Configuration.deprecation: > mapred.min.split.size.per.node is deprecated. Instead, use > mapreduce.input.fileinputformat.split.minsize.per.node > 14/02/26 05:42:35 INFO Configuration.deprecation: > mapred.min.split.size.per.rack is deprecated. Instead, use > mapreduce.input.fileinputformat.split.minsize.per.rack > 14/02/26 05:42:35 INFO Configuration.deprecation: mapred.max.split.size is > deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize > 14/02/26 05:42:35 INFO input.FileInputFormat: Total input paths to process : 1 > 14/02/26 05:42:35 INFO input.CombineFileInputFormat: DEBUG: Terminated node > allocation with : CompletedNodes: 1, size left: 0 > 14/02/26 05:42:35 INFO io.CombineHiveInputFormat: number of splits 1 > 14/02/26 05:42:35 INFO io.CombineHiveInputFormat: </PERFLOG method=getSplits > start=1393422155194 end=1393422155204 duration=10> > 14/02/26 05:42:35 INFO mapreduce.JobSubmitter: number of splits:1 > 14/02/26 05:42:35 INFO Configuration.deprecation: mapred.reduce.tasks is > deprecated. Instead, use mapreduce.job.reduces > 14/02/26 05:42:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: > job_1393416170595_0002 > 14/02/26 05:42:35 INFO impl.YarnClientImpl: Submitted application > application_1393416170595_0002 to ResourceManager at > sandbox.hortonworks.com/10.0.2.15:8050 > 14/02/26 05:42:35 INFO mapreduce.Job: The url to track the job: > http://sandbox.hortonworks.com:8088/proxy/application_1393416170595_0002/ > Starting Job = job_1393416170595_0002, Tracking URL = > http://sandbox.hortonworks.com:8088/proxy/application_1393416170595_0002/ > 14/02/26 05:42:35 INFO exec.Task: Starting Job = job_1393416170595_0002, > Tracking URL = > http://sandbox.hortonworks.com:8088/proxy/application_1393416170595_0002/ > Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1393416170595_0002 > 14/02/26 05:42:35 INFO exec.Task: Kill Command = /usr/lib/hadoop/bin/hadoop > job -kill job_1393416170595_0002 > Hadoop job information for Stage-0: number of mappers: 1; number of reducers: > 0 > 14/02/26 05:42:42 INFO exec.Task: Hadoop job information for Stage-0: number > of mappers: 1; number of reducers: 0 > 14/02/26 05:42:42 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > 2014-02-26 05:42:42,747 Stage-0 map = 0%, reduce = 0% > 14/02/26 05:42:42 INFO exec.Task: 2014-02-26 05:42:42,747 Stage-0 map = 0%, > reduce = 0% > 14/02/26 05:43:06 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > 2014-02-26 05:43:06,683 Stage-0 map = 100%, reduce = 0% > 14/02/26 05:43:06 INFO exec.Task: 2014-02-26 05:43:06,683 Stage-0 map = 100%, > reduce = 0% > 14/02/26 05:43:09 WARN mapreduce.Counters: Group > org.apache.hadoop.mapred.Task$Counter is deprecated. Use > org.apache.hadoop.mapreduce.TaskCounter instead > Ended Job = job_1393416170595_0002 with errors > 14/02/26 05:43:09 ERROR exec.Task: Ended Job = job_1393416170595_0002 with > errors > 14/02/26 05:43:09 INFO impl.YarnClientImpl: Killing application > application_1393416170595_0002 > 14/02/26 05:43:09 INFO ql.Driver: </PERFLOG method=task.MAPRED.Stage-0 > start=1393422154585 end=1393422189912 duration=35327> > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > 14/02/26 05:43:09 ERROR ql.Driver: FAILED: Execution Error, return code 2 > from org.apache.hadoop.hive.ql.exec.mr.MapRedTask > 14/02/26 05:43:09 INFO ql.Driver: </PERFLOG method=Driver.execute > start=1393422154583 end=1393422189912 duration=35329> > MapReduce Jobs Launched: > 14/02/26 05:43:09 INFO ql.Driver: MapReduce Jobs Launched: > 14/02/26 05:43:09 WARN mapreduce.Counters: Group FileSystemCounters is > deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead > Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > 14/02/26 05:43:09 INFO ql.Driver: Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 > FAIL > Total MapReduce CPU Time Spent: 0 msec > 14/02/26 05:43:09 INFO ql.Driver: Total MapReduce CPU Time Spent: 0 msec > 14/02/26 05:43:09 ERROR beeswax.BeeswaxServiceImpl: Exception while > processing query > BeeswaxException(message:Driver returned: 2. Errors: OK > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_1393416170595_0002, Tracking URL = > http://sandbox.hortonworks.com:8088/proxy/application_1393416170595_0002/ > Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1393416170595_0002 > Hadoop job information for Stage-0: number of mappers: 1; number of reducers: > 0 > 2014-02-26 05:42:42,747 Stage-0 map = 0%, reduce = 0% > 2014-02-26 05:43:06,683 Stage-0 map = 100%, reduce = 0% > Ended Job = job_1393416170595_0002 with errors > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > Total MapReduce CPU Time Spent: 0 msec > , log_context:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5, > handle:QueryHandle(id:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5, > log_context:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5), SQLState: ) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:351) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:609) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:598) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:337) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1471) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:598) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > 14/02/26 05:43:10 ERROR security.UserGroupInformation: > PriviledgedActionException as:hue (auth:SIMPLE) > cause:BeeswaxException(message:Driver returned: 2. Errors: OK > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_1393416170595_0002, Tracking URL = > http://sandbox.hortonworks.com:8088/proxy/application_1393416170595_0002/ > Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1393416170595_0002 > Hadoop job information for Stage-0: number of mappers: 1; number of reducers: > 0 > 2014-02-26 05:42:42,747 Stage-0 map = 0%, reduce = 0% > 2014-02-26 05:43:06,683 Stage-0 map = 100%, reduce = 0% > Ended Job = job_1393416170595_0002 with errors > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > Total MapReduce CPU Time Spent: 0 msec > , log_context:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5, > handle:QueryHandle(id:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5, > log_context:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5), SQLState: ) > 14/02/26 05:43:10 ERROR beeswax.BeeswaxServiceImpl: Caught unexpected > exception. > java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1504) > at > com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772) > at > com.cloudera.beeswax.BeeswaxServiceImpl.fetch(BeeswaxServiceImpl.java:980) > at > com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:987) > at > com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:971) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: BeeswaxException(message:Driver returned: 2. Errors: OK > Total MapReduce jobs = 1 > Launching Job 1 out of 1 > Number of reduce tasks is set to 0 since there's no reduce operator > Starting Job = job_1393416170595_0002, Tracking URL = > http://sandbox.hortonworks.com:8088/proxy/application_1393416170595_0002/ > Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1393416170595_0002 > Hadoop job information for Stage-0: number of mappers: 1; number of reducers: > 0 > 2014-02-26 05:42:42,747 Stage-0 map = 0%, reduce = 0% > 2014-02-26 05:43:06,683 Stage-0 map = 100%, reduce = 0% > Ended Job = job_1393416170595_0002 with errors > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.mr.MapRedTask > MapReduce Jobs Launched: > Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL > Total MapReduce CPU Time Spent: 0 msec > , log_context:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5, > handle:QueryHandle(id:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5, > log_context:aaf7e696-5cbe-4d4d-82ee-88dc6c7154d5), SQLState: ) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.execute(BeeswaxServiceImpl.java:351) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:609) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1$1.run(BeeswaxServiceImpl.java:598) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:337) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1471) > at > com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState$1.run(BeeswaxServiceImpl.java:598) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > > Thanks. Cheers, Yann -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9aa69a10-09f4-408c-87be-db1749485d6a%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
