Hi Furcy,
Thanks for the info. I ran the same job twice, killing it first time and starting again. Actually your point about 5 min duration seems to be correct. my process basically creates a new hive table with two additional columns and populate it from an existing table hence the locks Even if the job is killed “WAITING EXCLUSIVE” locks are still held on the new table (see below) and that is the cause of the issue Lock ID Database Table Partition State Type Transaction ID Last Hearbeat Acquired At User Hostname 14031 asehadoop t NULL WAITING EXCLUSIVE NULL 1445586539232 NULL hduser rhes564 14029 asehadoop t_staging NULL ACQUIRED SHARED_READ NULL 1445586247044 1445585940653 hduser rhes564 14029 asehadoop t NULL ACQUIRED SHARED_READ NULL 1445586247044 1445585940654 hduser rhes564 14030 asehadoop t NULL WAITING EXCLUSIVE NULL 1445586471827 NULL hduser rhes564 I am not sure this is behaving like classic RDBMS like Sybase or MSSQL where the rollback is happening after KILL command and the locks are held until rollback is complete. Killing a process itself will not release the locks! Regards, Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Furcy Pin [mailto:furcy....@flaminem.com] Sent: 23 October 2015 09:08 To: user@hive.apache.org Subject: Re: locks are held on tables even when no job running Hi Mich, I believe the duration of locks is defined by hive.txn.timeout, which is 5 min by default. https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties Retry your SHOW LOCKS command and check that the Last HeartBeat is not changing. If it is, it means your query is still active somehow. If it isn't, the lock should disappear by itself after the timeout. Also, I don't know if this is a known issue in Hive's Jira, but we noticed that trying to read and write data into the same table within a single query creates a deadlock. If that is what you were trying to do, you should rather write your data in a temporary file and then move it back into the table. On Fri, Oct 23, 2015 at 9:56 AM, Mich Talebzadeh <m...@peridale.co.uk <mailto:m...@peridale.co.uk> > wrote: Hi, What is the duration of locks held in Hive? I have got the following locks in Hive, although I have already killed the jobs! Lock ID Database Table Partition State Type Transaction ID Last Hearbeat Acquired At User Hostname 14031 asehadoop t NULL WAITING EXCLUSIVE NULL 1445586539232 NULL hduser rhes564 14029 asehadoop t_staging NULL ACQUIRED SHARED_READ NULL 1445586247044 1445585940653 hduser rhes564 14029 asehadoop t NULL ACQUIRED SHARED_READ NULL 1445586247044 1445585940654 hduser rhes564 14030 asehadoop t NULL WAITING EXCLUSIVE NULL 1445586471827 NULL hduser rhes564 mapred job -list Total jobs:0 JobId State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info No locks are held in metastore (Oracle in my case) as well. Thanks Mich Talebzadeh Sybase ASE 15 Gold Medal Award 2008 A Winning Strategy: Running the most Critical Financial Data on ASE 15 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", ISBN 978-0-9563693-0-7. co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 978-0-9759693-0-4 Publications due shortly: Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8 Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one out shortly http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
