[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14003011#comment-14003011 ] Lefty Leverenz commented on HIVE-5626: -- *hive.metastore.try.direct.sql.ddl* is now documented in the wiki: * [Configuration Properties - hive.metastore.try.direct.sql.ddl | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.try.direct.sql.ddl] enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Fix For: 0.13.0 Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933031#comment-13933031 ] Lefty Leverenz commented on HIVE-5626: -- *hive.metastore.try.direct.sql.ddl* needs a description in hive-default.xml.template (or in HiveConf.java after HIVE-6037 commits). [~sershe], if you write a release note for this jira I can make sure it gets into the appropriate file and add it to the wiki. enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Fix For: 0.13.0 Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13818165#comment-13818165 ] Ashutosh Chauhan commented on HIVE-5626: +1 enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817477#comment-13817477 ] Ashutosh Chauhan commented on HIVE-5626: Patch looks good. Thanks for refactoring. How did you test this one, by looking at logs ? Is it possible to add junit tests for this as we have added for direct-sql for other cases? enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817564#comment-13817564 ] Sergey Shelukhin commented on HIVE-5626: all the .q tests are ran with verifying object store which runs SQL and JDO and compares. Separate test - what do you mean? enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1381#comment-1381 ] Sergey Shelukhin commented on HIVE-5626: ping enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810486#comment-13810486 ] Sergey Shelukhin commented on HIVE-5626: I don't think the test failure is related. [~ashutoshc] wdyt? enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[
https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13810849#comment-13810849
]
Hive QA commented on HIVE-5626:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12611238/HIVE-5626.02.patch
{color:green}SUCCESS:{color} +1 4547 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/88/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/88/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
enable metastore direct SQL for drop/similar queries
Key: HIVE-5626
URL: https://issues.apache.org/jira/browse/HIVE-5626
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor
Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch
Metastore direct SQL is currently disabled for any queries running inside
external transaction (i.e. all modification queries, like dropping stuff).
This was done to keep the strictly performance-optimization behavior when
using Postgres, which unlike other RDBMS-es fails the tx on any syntax error;
so, if direct SQL is broken there's no way to fall back. So, it is disabled
for these cases.
It is not as important because drop commands are rare, but we might want to
address that. Either by some config setting or by making it work on
non-postgres DBs.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809526#comment-13809526 ] Ashutosh Chauhan commented on HIVE-5626: If we have config enabled by default, then will postgres be broken or will it choose slow datanucleus path. If postgres chooses slow DN path, will that be for all queries or only for drop queries? enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809578#comment-13809578 ] Sergey Shelukhin commented on HIVE-5626: this only affects drop and similar queries. Assuming direct SQL works (which it should), it will use it. But if there's some issue, it won't be able to fall back to DN for these queries, so they would fail. Then this setting would need to be disabled for it to work via the slow DN path. enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[
https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13809897#comment-13809897
]
Hive QA commented on HIVE-5626:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12610938/HIVE-5626.01.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4489 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.listener.TestNotificationListener.testAMQListener
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/38/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/38/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
enable metastore direct SQL for drop/similar queries
Key: HIVE-5626
URL: https://issues.apache.org/jira/browse/HIVE-5626
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor
Attachments: HIVE-5626.01.patch, HIVE-5626.02.patch, HIVE-5626.patch
Metastore direct SQL is currently disabled for any queries running inside
external transaction (i.e. all modification queries, like dropping stuff).
This was done to keep the strictly performance-optimization behavior when
using Postgres, which unlike other RDBMS-es fails the tx on any syntax error;
so, if direct SQL is broken there's no way to fall back. So, it is disabled
for these cases.
It is not as important because drop commands are rare, but we might want to
address that. Either by some config setting or by making it work on
non-postgres DBs.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13808558#comment-13808558 ] Sergey Shelukhin commented on HIVE-5626: https://reviews.apache.org/r/15067/ enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Priority: Minor Attachments: HIVE-5626.01.patch, HIVE-5626.patch Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[
https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13807604#comment-13807604
]
Hive QA commented on HIVE-5626:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12610719/HIVE-5626.patch
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 4512 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testListFSRoot
org.apache.hadoop.hive.ql.history.TestHiveHistory.testQueryloglocParentDirNotExist
org.apache.hadoop.hive.ql.metadata.TestHive.testTable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testTable
org.apache.hcatalog.api.TestHCatClient.testBasicDDLCommands
org.apache.hcatalog.cli.TestPermsGrp.testCustomPerms
org.apache.hcatalog.hbase.TestHBaseHCatStorageHandler.testTableCreateDrop
org.apache.hcatalog.hbase.TestHBaseHCatStorageHandler.testTableCreateDropDifferentCase
org.apache.hcatalog.hbase.TestHCatHBaseInputFormat.TestHBaseTableReadMR
org.apache.hcatalog.hbase.TestSnapshots.TestSnapshotConversion
org.apache.hcatalog.hbase.snapshot.TestZNodeSetUp.testBasicZNodeCreation
org.apache.hcatalog.mapreduce.TestHCatInputFormat.testBadRecordHandlingFails
org.apache.hcatalog.mapreduce.TestHCatNonPartitioned.testHCatNonPartitionedTable
org.apache.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hcatalog.pig.TestHCatLoaderStorer.testSmallTinyInt
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps
org.apache.hive.hcatalog.hbase.TestHBaseInputFormat.TestHBaseTableReadMR
org.apache.hive.hcatalog.hbase.TestHiveHBaseStorageHandler.testTableCreateDrop
org.apache.hive.hcatalog.hbase.TestHiveHBaseStorageHandler.testTableCreateDropDifferentCase
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.testPigPopulation
org.apache.hive.hcatalog.mapreduce.TestHCatExternalHCatNonPartitioned.testHCatNonPartitionedTable
org.apache.hive.hcatalog.pig.TestOrcHCatLoaderComplexSchema.testSyntheticComplexSchema
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testConfOverlay
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatement
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatement
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1274/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1274/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
enable metastore direct SQL for drop/similar queries
Key: HIVE-5626
URL: https://issues.apache.org/jira/browse/HIVE-5626
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Priority: Minor
Attachments: HIVE-5626.patch
Metastore direct SQL is currently disabled for any queries running inside
external transaction (i.e. all modification queries, like dropping stuff).
This was done to keep the strictly performance-optimization behavior when
using Postgres, which unlike other RDBMS-es fails the tx on any syntax error;
so, if direct SQL is broken there's no way to fall back. So, it is disabled
for these cases.
It is not as important because drop commands are rare, but we might want to
address that. Either by some config setting or by making it work on
non-postgres DBs.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HIVE-5626) enable metastore direct SQL for drop/similar queries
[ https://issues.apache.org/jira/browse/HIVE-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803119#comment-13803119 ] Sergey Shelukhin commented on HIVE-5626: [~ashutoshc] fyi enable metastore direct SQL for drop/similar queries Key: HIVE-5626 URL: https://issues.apache.org/jira/browse/HIVE-5626 Project: Hive Issue Type: Improvement Reporter: Sergey Shelukhin Priority: Minor Metastore direct SQL is currently disabled for any queries running inside external transaction (i.e. all modification queries, like dropping stuff). This was done to keep the strictly performance-optimization behavior when using Postgres, which unlike other RDBMS-es fails the tx on any syntax error; so, if direct SQL is broken there's no way to fall back. So, it is disabled for these cases. It is not as important because drop commands are rare, but we might want to address that. Either by some config setting or by making it work on non-postgres DBs. -- This message was sent by Atlassian JIRA (v6.1#6144)
