The approach is similar, but it is not identical. Let me go over the query based compaction codepath to see if I spot this bug there.
Thanks, --Vaibhav From: Aditya Shah <adityashah3...@gmail.com> Date: Saturday, February 16, 2019 at 3:44 AM To: Vaibhav Gumashta <vgumas...@hortonworks.com> Cc: "dev@hive.apache.org" <dev@hive.apache.org>, Eugene Koifman <ekoif...@hortonworks.com>, Gopal Vijayaraghavan <go...@hortonworks.com> Subject: Re: Null pointer exception on running compaction against an MM table [mage removed by sender.] Hi, Thanks for the reply, have opened a JIRA (HIVE-21280) for the same and will upload a patch soon. But I further had doubts on the new query based compactor for full CRUD tables that has gone into master in HIVE-20699. Does major compaction work there using query based compactor similar to the one for MM table, because I expect the same problem to exist there as well? Aditya On Sat, Feb 16, 2019 at 2:34 AM Vaibhav Gumashta <vgumas...@hortonworks.com<mailto:vgumas...@hortonworks.com>> wrote: Aditya, Thanks for reporting this. Would you like to create a jira for this (https://issues.apache.org/jira/projects/HIVE)? Additionally, if you would like to work on a fix, I’m happy to help in reviewing. --Vaibhav From: Aditya Shah <adityashah3...@gmail.com<mailto:adityashah3...@gmail.com>> Date: Friday, February 15, 2019 at 2:05 AM To: "dev@hive.apache.org<mailto:dev@hive.apache.org>" <dev@hive.apache.org<mailto:dev@hive.apache.org>> Cc: Eugene Koifman <ekoif...@hortonworks.com<mailto:ekoif...@hortonworks.com>>, Vaibhav Gumashta <vgumas...@hortonworks.com<mailto:vgumas...@hortonworks.com>>, Gopal Vijayaraghavan <go...@hortonworks.com<mailto:go...@hortonworks.com>> Subject: Null pointer exception on running compaction against an MM table Error! Filename not specified. Hi, I was trying to run compaction on MM table but got a null pointer exception while getting HDFS session path. The error seemed to me that session state was not started for this queries. Am I missing something here? I do think session state needs to be started for each of the queries (insert into temp table etc) running for compaction (I'm also doubtful for statsupdater thread's queries) on HMS. Some details are as follows: Env./Versions: Using Hive-3.1.1 (rel/release-3.1.1) Steps to reproduce: 1) Using beeline with HS2 and HMS 2) create an MM table 3) Insert a few values in the table 4) alter table mm_table compact 'major' and wait; Stack trace on HMS: compactor.Worker: Caught exception while trying to compact id:8,dbname:default,tableName:acid_mm_orc,partName:null,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestWriteId:0. Marking failed to avoid repeated failures, java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run create temporary table default.tmp_compactor_acid_mm_orc_1550222367257(`a` int, `b` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'WITH SERDEPROPERTIES ( 'serialization.format'='1')STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://localhost:9000/user/hive/warehouse/acid_mm_orc/_tmp_2d8a096c-2db5-4ed8-921c-b3f6d31e079e/_base' TBLPROPERTIES ('transactional'='false') at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runMmCompaction(CompactorMR.java:373) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:241) at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run create temporary table default.tmp_compactor_acid_mm_orc_1550222367257(`a` int, `b` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'WITH SERDEPROPERTIES ( 'serialization.format'='1')STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://localhost:9000/user/hive/warehouse/acid_mm_orc/_tmp_2d8a096c-2db5-4ed8-921c-b3f6d31e079e/_base' TBLPROPERTIES ('transactional'='false') at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runOnDriver(CompactorMR.java:525) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runMmCompaction(CompactorMR.java:365) ... 2 more Caused by: java.lang.NullPointerException: Non-local session path expected to be non-null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:228) at org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:815) at org.apache.hadoop.hive.ql.Context.<init>(Context.java:309) at org.apache.hadoop.hive.ql.Context.<init>(Context.java:295) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:591) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1684) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1807) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1567) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1556) at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runOnDriver(CompactorMR.java:522) ... 3 more Observations: 1) SessionState.start() initializes paths, hivehist etc. 2) SessionState is not started in setupSessionState() in runMMCompaction(). (There is also a comment by Sergey in the code regarding the same) 3) Even after making it start the session state it further fails in running a Teztask for insert overwrite on temp table with the contents of the original table. 4) The cause for 3) is Tezsession state is not able to initialize due to Illegal Argument exception being thrown at the time of setting up caller context in Tez task due to caller id being empty 5) Reason for 4) is queryid is an empty string for such queries. 6) A possible solution for 5) Building querystate with queryid in runOnDriver() in DriverUtils.java Do let me know if you need some more information for the same. Thanks and Regards, Aditya Shah 5th Year M.Sc.(Hons.) Mathematics & B.E.(Hons.) Computer Science and Engineering ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ Birla Institute of Technology & Science, Pilani Vidhya Vihar Pilani 333 031(Raj.), India Phone:- +91 7689996342 BITS Mail<https://mail.google.com/mail/u/0/#inbox> -- Aditya Shah 4th Year M.Sc.(Hons.) Mathematics & B.E.(Hons.) Computer Science and Engineering ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ Birla Institute of Technology & Science, Pilani Vidhya Vihar Pilani 333 031(Raj.), India Phone:- Error! Filename not specified.+91 7689996342 BITS Mail<https://mail.google.com/mail/u/0/#inbox>
