Aditya,
Thanks for reporting this. Would you like to create a jira for this
(https://issues.apache.org/jira/projects/HIVE)? Additionally, if you would like
to work on a fix, I’m happy to help in reviewing.
--Vaibhav
From: Aditya Shah <adityashah3...@gmail.com>
Date: Friday, February 15, 2019 at 2:05 AM
To: "dev@hive.apache.org" <dev@hive.apache.org>
Cc: Eugene Koifman <ekoif...@hortonworks.com>, Vaibhav Gumashta
<vgumas...@hortonworks.com>, Gopal Vijayaraghavan <go...@hortonworks.com>
Subject: Null pointer exception on running compaction against an MM table
[mage removed by sender.]
Hi,
I was trying to run compaction on MM table but got a null pointer exception
while getting HDFS session path. The error seemed to me that session state was
not started for this queries. Am I missing something here? I do think session
state needs to be started for each of the queries (insert into temp table etc)
running for compaction (I'm also doubtful for statsupdater thread's queries) on
HMS. Some details are as follows:
Env./Versions: Using Hive-3.1.1 (rel/release-3.1.1)
Steps to reproduce:
1) Using beeline with HS2 and HMS
2) create an MM table
3) Insert a few values in the table
4) alter table mm_table compact 'major' and wait;
Stack trace on HMS:
compactor.Worker: Caught exception while trying to compact
id:8,dbname:default,tableName:acid_mm_orc,partName:null,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestWriteId:0.
Marking failed to avoid repeated failures, java.io.IOException:
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run create
temporary table default.tmp_compactor_acid_mm_orc_1550222367257(`a` int, `b`
string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'WITH
SERDEPROPERTIES (
'serialization.format'='1')STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION
'hdfs://localhost:9000/user/hive/warehouse/acid_mm_orc/_tmp_2d8a096c-2db5-4ed8-921c-b3f6d31e079e/_base'
TBLPROPERTIES ('transactional'='false')
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runMmCompaction(CompactorMR.java:373)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:241)
at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:174)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to run
create temporary table default.tmp_compactor_acid_mm_orc_1550222367257(`a` int,
`b` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'WITH
SERDEPROPERTIES (
'serialization.format'='1')STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION
'hdfs://localhost:9000/user/hive/warehouse/acid_mm_orc/_tmp_2d8a096c-2db5-4ed8-921c-b3f6d31e079e/_base'
TBLPROPERTIES ('transactional'='false')
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runOnDriver(CompactorMR.java:525)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runMmCompaction(CompactorMR.java:365)
... 2 more
Caused by: java.lang.NullPointerException: Non-local session path expected to
be non-null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:228)
at
org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:815)
at org.apache.hadoop.hive.ql.Context.<init>(Context.java:309)
at org.apache.hadoop.hive.ql.Context.<init>(Context.java:295)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:591)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1684)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1807)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1567)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1556)
at
org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.runOnDriver(CompactorMR.java:522)
... 3 more
Observations: 1) SessionState.start() initializes paths, hivehist etc.
2) SessionState is not started in setupSessionState() in runMMCompaction().
(There is also a comment by Sergey in the code regarding the same)
3) Even after making it start the session state it further fails in running a
Teztask for insert overwrite on temp table with the contents of the original
table.
4) The cause for 3) is Tezsession state is not able to initialize due to
Illegal Argument exception being thrown at the time of setting up caller
context in Tez task due to caller id being empty
5) Reason for 4) is queryid is an empty string for such queries.
6) A possible solution for 5) Building querystate with queryid in
runOnDriver() in DriverUtils.java
Do let me know if you need some more information for the same.
Thanks and Regards,
Aditya Shah
5th Year
M.Sc.(Hons.) Mathematics & B.E.(Hons.) Computer Science and Engineering
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
Birla Institute of Technology & Science, Pilani
Vidhya Vihar
Pilani 333 031(Raj.), India
Phone:- +91 7689996342
BITS Mail<https://mail.google.com/mail/u/0/#inbox>
