[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467186#comment-16467186
]
Peter Vary commented on HIVE-18906:
---
+1
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467006#comment-16467006
]
Antal Sinkovits commented on HIVE-18906:
The test failures are not related.
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464616#comment-16464616
]
Hive QA commented on HIVE-18906:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12921913/HIVE-18906.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 14319 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_time_window]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_dynpart_hashjoin_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_stats]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomReadOps
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesWrite
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=241)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerVertexRawInputSplitsNoKill
(batchId=241)
org.apache.hive.spark.client.rpc.TestRpc.testServerPort (batchId=304)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10699/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10699/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10699/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 37 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12921913 - PreCommit-HIVE-Build
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hiv
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464587#comment-16464587
]
Hive QA commented on HIVE-18906:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10699/dev-support/hive-personality.sh
|
| git revision | master / 52f1b24 |
| Default Java | 1.8.0_111 |
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10699/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-18906.1.patch
>
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB i
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462970#comment-16462970
]
BELUGA BEHR commented on HIVE-18906:
It's kinda of a weird test. I'm not sure why that test clears a RDBMS for
direct-SQL. Oh well.
But I think you understand the basic idea of what I'm asking for here? Less
logging?
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462273#comment-16462273
]
Antal Sinkovits commented on HIVE-18906:
Thanks for the answers [~belugabehr]
1. The current implementation executes a test query, to determine whether it
supports direct sql or not. See:
https://github.com/apache/hive/blob/master/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L182
Not sure why it was implemented that way, it might be a good idea, to use the
database type instead, but I think we shouldn't mix the two, as it creates
ambiguity.
As this enhancement might be a good idea (even enhance performance as well), I
do feel, that its over the scope of this jira.
2. It seems so to me. This is what the doc says, and the related jira. I've
also did a bit of a smoke test, by running two beeline, changing the config in
one of them and checking the value in both.
I will use the described approach.
Docs/jira/commands:
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.try.direct.sql
https://issues.apache.org/jira/browse/HIVE-7532
set metaconf:hive.metastore.try.direct.sql=false;
set metaconf:hive.metastore.try.direct.sql;
3. Yes this makes sense, but in this case, we have the same issue as in 1.
4. Ok, this is also fine then.
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461535#comment-16461535
]
BELUGA BEHR commented on HIVE-18906:
Thank you [~asinkovits] for your interest
# We should be able to determine the database type from the JDBC connection
string (jdbc:mysql://) or perhaps from the driver class definition
(com.mysql.jdbc.Driver).. This would not require us to make a connection to
determine the database type. In Cloudera Manager, there is a configuration
"Hive Metastore Database Type" that tells us the vendor information without
making a connection.
# Are you sure this can be set at the session level? This is a HMS
optimization, not HS2. That would be fine if it's logged at the session level
if required.
# No, there should only be a warning for the cluster administrator if the
service is configured incorrectly from the get-go. If it's possible to
override this per-session, then there's probably a reason for it and I don't
need to be told it's a problem. There must already be a known problem if I
have to disable it manually.
# I think a _debug_ message telling me that the optimization is being used is
appropriate and then providing the actual SQL statement as _trace_ logging is
helpful if I require additional, deeper, troubleshooting.
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461358#comment-16461358
]
Antal Sinkovits commented on HIVE-18906:
"An INFO message at startup which tells the administrator if the back-end
database is supported for the direct SQL optimization"
It seems to me, that by design, there are no db connection at startup, so I
think this would introduce an overhead just for logging.
Is this really necessary?
"An INFO message at startup which tells the administrator if the direct SQL
optimization is enabled/disabled"
This is fine, and since this param can be overridden at session level, I would
suggest that we should print it there as well.
"A WARN message if the SQL direct optimization is disabled for a database that
is supported"
We can do this, for each query executed, although I do feel that its a bit
verbose. Do we want this?
"A DEBUG message for each query if it is using (or not using) direct SQL
optimization"
The concrete sql (when used) is logged at trace level. Do we need this?
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18906) Lower Logging for "Using direct SQL"
[
https://issues.apache.org/jira/browse/HIVE-18906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434438#comment-16434438
]
BELUGA BEHR commented on HIVE-18906:
I think what we want here is:
* An INFO message at startup which tells the administrator if the back-end
database is supported for the direct SQL optimization
* An INFO message at startup which tells the administrator if the direct SQL
optimization is enabled/disabled
* A WARN message if the SQL direct optimization is disabled for a database
that is supported
* A DEBUG message for each query if it is using (or not using) direct SQL
optimization
> Lower Logging for "Using direct SQL"
>
>
> Key: HIVE-18906
> URL: https://issues.apache.org/jira/browse/HIVE-18906
> Project: Hive
> Issue Type: Improvement
>Reporter: BELUGA BEHR
>Assignee: Antal Sinkovits
>Priority: Trivial
> Labels: noob
>
> https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L181
> {code}
> 2018-02-26 14:32:50,620 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:09,566 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:43,170 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:33:46,301 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:16,966 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:18,072 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-196]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:25,064 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:34:51,781 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:04,685 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-199]:
> Using direct SQL, underlying DB is MYSQL
> 2018-02-26 14:35:26,689 INFO
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql: [pool-4-thread-172]:
> Using direct SQL, underlying DB is MYSQL
> {code}
> I wouldn't mind knowing this information when the service starts up and
> detects that there is a Direct SQL compatible database, but I don't need to
> know about this every time.
> Please lower to _debug_ level logging and parameterize with SLF4J. Direct
> SQL is a performance thing and not something I care about during the normal
> course of operation (unless there is an error which would presumably be
> covered by an _error_ logging message).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
