[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546207#comment-16546207 ] Igor Kryvenko edited comment on HIVE-17683 at 7/17/18 9:26 AM: --- Seems like failure is not related {code:java} {code} I will attach the same patch to check this. Locally it also works fine: {code} [INFO] Running org.apache.hive.beeline.TestBeeLineWithArgs [WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 237.004 s - in org.apache.hive.beeline.TestBeeLineWithArgs [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 54, Failures: 0, Errors: 0, Skipped: 2 [INFO] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 05:00 min [INFO] Finished at: 2018-07-17T12:25:43+03:00 [INFO] Final Memory: 136M/2524M [INFO] {code} was (Author: ikryvenko): Seems like failure is not related {code} {code} I will attach the same patch to check this. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch, > HIVE-17683.03.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542380#comment-16542380 ] Eugene Koifman edited comment on HIVE-17683 at 7/13/18 12:54 AM: - [~ikryvenko], sorry, it took a while to get back to this. Your implementation creates ExplainTask.getJsonLocks() which duplicates a lot of the logic in DbTxnManger.acquireLocks(). This is problematic because they have to be kept in sync. Could you refactor it so that they share code? For example, create a {{LockRequest makeLockRequest(List, List)}} and use it in both places? Also, the refactoring in acquireLocks() lost {noformat} default: throw new IllegalArgumentException(String .format("Lock type [%s] for Database.Table [%s.%s] is unknown", lockType, t.getDbName(), t.getTableName() ));{noformat} This may change how errors are surfaced - not sure it's a good idea. Don't know if it's related to your changes but in explain_locks.q.out {{explain locks drop table test_explain_locks}} doesn't acquire any locks - this is odd - I'd expect X lock on the table for a drop command. Why did you chose to output the data as JSON? was (Author: ekoifman): [~ikryvenko], sorry, it took a while to get back to this. Your implementation creates ExplainTask.getJsonLocks() which duplicates a lot of the logic in DbTxnManger.acquireLocks(). This is problematic because they have to be kept in sync. Could you refactor it so that they share code? For example, create a {{LockRequest makeLockRequest(List, List)}} and use it in both places? Also, the refactoring in acquireLocks() lost {noformat} default: throw new IllegalArgumentException(String .format("Lock type [%s] for Database.Table [%s.%s] is unknown", lockType, t.getDbName(), t.getTableName() ));{noformat} This may change how errors are surfaced - not sure it's a good idea. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505119#comment-16505119 ] Igor Kryvenko edited comment on HIVE-17683 at 6/7/18 7:06 PM: -- {code} [ERROR] /data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22] cannot find symbol symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType) location: variable compBuilder of type org.apache.hadoop.hive.metastore.LockComponentBuilder {code} It can't find added method. Is the last patch good for you? was (Author: ikryvenko): {code} [ERROR] /data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22] cannot find symbol symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType) location: variable compBuilder of type org.apache.hadoop.hive.metastore.LockComponentBuilder {code} It can't find added method. > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505119#comment-16505119 ] Igor Kryvenko edited comment on HIVE-17683 at 6/7/18 7:06 PM: -- {code} [ERROR] /data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22] cannot find symbol symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType) location: variable compBuilder of type org.apache.hadoop.hive.metastore.LockComponentBuilder {code} It can't find added method. was (Author: ikryvenko): {code} [ERROR] /data/hiveptest/working/yetus_PreCommit-HIVE-Build-11593/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java:[521,22] cannot find symbol symbol: method setLock(org.apache.hadoop.hive.metastore.api.LockType) location: variable compBuilder of type org.apache.hadoop.hive.metastore.LockComponentBuilder {code} > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)