[jira] [Commented] (HIVE-17729) Add Database & Explain related blobstore tests

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195584#comment-16195584
 ] 

Hive QA commented on HIVE-17729:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890799/HIVE-17729.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11193 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=239)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7172/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7172/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7172/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890799 - PreCommit-HIVE-Build

> Add Database & Explain related blobstore tests
> --
>
> Key: HIVE-17729
> URL: https://issues.apache.org/jira/browse/HIVE-17729
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Rentao Wu
>Assignee: Rentao Wu
> Attachments: HIVE-17729.patch
>
>
> This patch introduces the following regression tests into the hive-blobstore 
> qtest module:
> * create_database.q  -> tests tables with location inherited from database
> * multiple_db.q  -> tests query spanning multiple databases
> * explain.q -> tests EXPLAIN INSERT OVERWRITE command
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17371) Move tokenstores to metastore module

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195571#comment-16195571
 ] 

Hive QA commented on HIVE-17371:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890793/HIVE-17371.03.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
 (batchId=242)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[binary_output_format] 
(batchId=84)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
 (batchId=240)
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
 (batchId=240)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7171/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7171/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7171/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890793 - PreCommit-HIVE-Build

> Move tokenstores to metastore module
> 
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch, HIVE-17371.02.patch, 
> HIVE-17371.03.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and 
> {{ZKTokenStore}} since they implement 
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of  
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws 
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf, 
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing 
> in a HiveConf for
> // configuration.  It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its 
> construction.  But this is
> // the hive-shims version of the memory store.  We want to convert this 
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
> 
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName)) 
> {
>   return new MemoryTokenStore();
> }
> try {
>   Class storeClass =
>   
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
>   return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
>   throw new IOException("Error initializing delegation token store: " + 
> tokenStoreClassName, e);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17701) Added restriction to historic queries on web UI

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195557#comment-16195557
 ] 

Hive QA commented on HIVE-17701:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890824/HIVE-17701.6.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=239)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning 
(batchId=291)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout 
(batchId=228)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7170/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7170/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7170/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890824 - PreCommit-HIVE-Build

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch, HIVE-17701.5.patch, HIVE-17701.6.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17728) TestHCatClient should use hive.metastore.transactional.event.listeners as per recommendation.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-17728:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> TestHCatClient should use hive.metastore.transactional.event.listeners as per 
> recommendation.
> -
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17728) TestHCatClient should use hive.metastore.transactional.event.listeners as per recommendation.

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195543#comment-16195543
 ] 

Sankar Hariappan commented on HIVE-17728:
-

Thanks for the review [~thejas]!
The patch is committed to master.

> TestHCatClient should use hive.metastore.transactional.event.listeners as per 
> recommendation.
> -
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17728) TestHCatClient should use hive.metastore.transactional.event.listeners as per recommendation.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-17728:

Summary: TestHCatClient should use 
hive.metastore.transactional.event.listeners as per recommendation.  (was: 
TestHCatClient should configure hive.metastore.transactional.event.listeners as 
per recommendation.)

> TestHCatClient should use hive.metastore.transactional.event.listeners as per 
> recommendation.
> -
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195512#comment-16195512
 ] 

Hive QA commented on HIVE-17728:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890776/HIVE-17728.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7169/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7169/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7169/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890776 - PreCommit-HIVE-Build

> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17566) Create schema required for workload management.

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17566:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the patch!

> Create schema required for workload management.
> ---
>
> Key: HIVE-17566
> URL: https://issues.apache.org/jira/browse/HIVE-17566
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Fix For: 3.0.0
>
> Attachments: HIVE-17566.01.patch, HIVE-17566.02.patch, 
> HIVE-17566.03.patch, HIVE-17566.04.patch, HIVE-17566.05.patch, 
> HIVE-17566.06.patch, HIVE-17566.07.patch
>
>
> Schema + model changes required for workload management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17724) Cast from string to float gives strange result

[Yuzhou Sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195490#comment-16195490
 ] 

Yuzhou Sun commented on HIVE-17724:
---

+1 think it's a limitation of the type

According to 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types and 
https://en.wikipedia.org/wiki/Single-precision_floating-point_format , FLOAT is 
4-byte single precision floating point number, which uses 8 bits for the 
exponent

Transfer the numbers to binary number:
1507284611 -> 101100111010111010101101011
1507284608 -> 101100111010111010101101000


> Cast from string to float gives strange result
> --
>
> Key: HIVE-17724
> URL: https://issues.apache.org/jira/browse/HIVE-17724
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
> Environment: Hive 1.1.0-cdh5.10.0
> Subversion 
> file:///data0/jenkins/workspace/generic-package-ubuntu64-14-04/CDH5.10.0-Packaging-Hive-2017-01-20_11-52-43/hive-1.1.0+cdh5.10.0+859-1.cdh5.10.0.p0.71~trusty
>  -r Unknown
>Reporter: David Scarlatti
>
> this query:"select CAST ('1507284611' as int), CAST ('1507284611' as bigint), 
>  CAST ('1507284611' as float),   CAST ('1507284611' as double),   CAST 
> ('1507284611' as decimal)"
> returns
> 15072846111507284611  1507284608  1507284611  1507284611
> cast as float is the only one changing the numeric value.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17736) ObjectStore transaction handling can be simplified

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov reassigned HIVE-17736:
-


> ObjectStore transaction handling can be simplified
> --
>
> Key: HIVE-17736
> URL: https://issues.apache.org/jira/browse/HIVE-17736
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> There are many places in ObjectStore that do something like this:
> {code}
> boolean commited = false;
>try {
>   openTransaction();
>   commited = commitTransaction();
> } finally {
>   if (!commited) {
> rollbackTransaction();
>   }
> }
> {code}
> We can simplify this in two ways:
> 1) Create a wrapper that calls given piece of code inside the block of code 
> above. This is similar to TransactionManager in Sentry.
> 2) Create a special auto-closeable object that does the check and rollback on 
> close.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17718) HS2 Logs print unnecessary stack trace when HoS query is cancelled

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195487#comment-16195487
 ] 

Hive QA commented on HIVE-17718:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890772/HIVE-17718.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7167/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7167/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7167/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890772 - PreCommit-HIVE-Build

> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17718.1.patch, HIVE-17718.2.patch, 
> HIVE-17718.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR 
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: 
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with 
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN  org.apache.hadoop.hive.ql.Driver: 
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR 
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: 
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with 
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> 

[jira] [Work started] (HIVE-11266) count(*) wrong result based on table statistics for external tables

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-11266 started by Jesus Camacho Rodriguez.
--
> count(*) wrong result based on table statistics for external tables
> ---
>
> Key: HIVE-11266
> URL: https://issues.apache.org/jira/browse/HIVE-11266
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Simone Battaglia
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-11266.patch
>
>
> Hive returns wrong count result on an external table with table statistics if 
> I change table data files.
> This is the scenario in details:
> 1) create external table my_table (...) location 'my_location';
> 2) analyze table my_table compute statistics;
> 3) change/add/delete one or more files in 'my_location' directory;
> 4) select count(\*) from my_table;
> In this case the count query doesn't generate a MR job and returns the result 
> based on table statistics. This result is wrong because is based on 
> statistics stored in the Hive metastore and doesn't take into account 
> modifications introduced on data files.
> Obviously setting "hive.compute.query.using.stats" to FALSE this problem 
> doesn't occur but the default value of this property is TRUE.
> I thinks that also this post on stackoverflow, that shows another type of bug 
> in case of multiple insert, is related to the one that I reported:
> http://stackoverflow.com/questions/24080276/wrong-result-for-count-in-hive-table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-11266) count(*) wrong result based on table statistics for external tables

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-11266:
---
Attachment: HIVE-11266.patch

> count(*) wrong result based on table statistics for external tables
> ---
>
> Key: HIVE-11266
> URL: https://issues.apache.org/jira/browse/HIVE-11266
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Simone Battaglia
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-11266.patch
>
>
> Hive returns wrong count result on an external table with table statistics if 
> I change table data files.
> This is the scenario in details:
> 1) create external table my_table (...) location 'my_location';
> 2) analyze table my_table compute statistics;
> 3) change/add/delete one or more files in 'my_location' directory;
> 4) select count(\*) from my_table;
> In this case the count query doesn't generate a MR job and returns the result 
> based on table statistics. This result is wrong because is based on 
> statistics stored in the Hive metastore and doesn't take into account 
> modifications introduced on data files.
> Obviously setting "hive.compute.query.using.stats" to FALSE this problem 
> doesn't occur but the default value of this property is TRUE.
> I thinks that also this post on stackoverflow, that shows another type of bug 
> in case of multiple insert, is related to the one that I reported:
> http://stackoverflow.com/questions/24080276/wrong-result-for-count-in-hive-table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-11266) count(*) wrong result based on table statistics for external tables

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-11266:
---
Status: Patch Available  (was: In Progress)

> count(*) wrong result based on table statistics for external tables
> ---
>
> Key: HIVE-11266
> URL: https://issues.apache.org/jira/browse/HIVE-11266
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Simone Battaglia
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Attachments: HIVE-11266.patch
>
>
> Hive returns wrong count result on an external table with table statistics if 
> I change table data files.
> This is the scenario in details:
> 1) create external table my_table (...) location 'my_location';
> 2) analyze table my_table compute statistics;
> 3) change/add/delete one or more files in 'my_location' directory;
> 4) select count(\*) from my_table;
> In this case the count query doesn't generate a MR job and returns the result 
> based on table statistics. This result is wrong because is based on 
> statistics stored in the Hive metastore and doesn't take into account 
> modifications introduced on data files.
> Obviously setting "hive.compute.query.using.stats" to FALSE this problem 
> doesn't occur but the default value of this property is TRUE.
> I thinks that also this post on stackoverflow, that shows another type of bug 
> in case of multiple insert, is related to the one that I reported:
> http://stackoverflow.com/questions/24080276/wrong-result-for-count-in-hive-table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-9350) Add ability for HiveAuthorizer implementations to filter out results of 'show tables', 'show databases'

[Alexander Kolbasov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195480#comment-16195480
 ] 

Alexander Kolbasov commented on HIVE-9350:
--

[~thejas] What was the reason for adding {{MetaException}} and 
{{NoSuchObjectException}} to {{MetaStoreFilterHook}} interfaces? Looks like the 
only implementation isn't throwing any exceptions.

The problem is that the change requires clients to recompile against the new 
interface. 


> Add ability for HiveAuthorizer implementations to filter out results of 'show 
> tables', 'show databases'
> ---
>
> Key: HIVE-9350
> URL: https://issues.apache.org/jira/browse/HIVE-9350
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>  Labels: TODOC1.2
> Fix For: 1.2.0
>
> Attachments: HIVE-9350.1.patch, HIVE-9350.2.patch, HIVE-9350.3.patch, 
> HIVE-9350.4.patch, HIVE-9350.5.patch
>
>
> It should be possible for HiveAuthorizer implementations to control if a user 
> is able to see a table or database in results of 'show tables' and 'show 
> databases' respectively.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17704) Hive QA isn't collecting spark.log anymore

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195479#comment-16195479
 ] 

Sahil Takiar commented on HIVE-17704:
-

Ok had some time to work on this today, here is what I found. The {{spark.log}} 
file isn't generated locally anymore. I went back to branch-1.1 and couldn't 
even it generate it locally there either. On branch-1.1, the file will only be 
generated if Spark picks up {{data/conf/spark/log4j.properties}} as the log4j 
config file for the Driver and Executors. Hive tries to ensure this happens by 
copying the config file into {{SPARK_HOME/conf/}}, but as far as I can tell 
that doesn't always work. I used the {{-Dlog4j.debug}} and found out that Spark 
will usually pick up the log4j file from some jar on its classpath. Most blog 
posts online discussing how to configure log4j for Spark require setting 
{{-Dlog4j.configuration=}} in the {{spark.driver.extraJavaOptions}} config. I 
tried that locally at it seems to work. I think there are a few other fixes 
that require this to work for the master branch. For one, Spark is still on 
log4j1.x, but the properties file we are giving it is for log4j2.x. It would be 
nice if each executor logged to a different file.

Alternatively, we could just archive everything under 
{{itests/qtest-spark/target/sparkOnYarn/sparkOnYarn-logDir}} which contains all 
the container logs. Which is probably the easiest approach

> Hive QA isn't collecting spark.log anymore
> --
>
> Key: HIVE-17704
> URL: https://issues.apache.org/jira/browse/HIVE-17704
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>
> Checked a recent Hive QA run, and the {{spark.log}} file isn't being 
> collected by ptest (maybe isn't not even being generated anymore, haven't 
> checked).
> We should re-enable it, its hard to debug Hive QA runs without the 
> {{spark.log}} file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries

[Alexander Kolbasov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195464#comment-16195464
 ] 

Alexander Kolbasov commented on HIVE-17735:
---

[~anishek], [~spena] FYI

> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>
> In ObjectStore.addNotificationEvent():
> {code}
>   Query objectQuery = pm.newQuery(MNotificationNextId.class);
>   Collection ids = (Collection) 
> objectQuery.execute();
> {code}
> The query is never closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries

[Alexander Kolbasov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195463#comment-16195463
 ] 

Alexander Kolbasov commented on HIVE-17735:
---

There is a second leak:

{code}
  private void lockForUpdate() throws MetaException {
String selectQuery = "select NEXT_EVENT_ID from NOTIFICATION_SEQUENCE";
String selectForUpdateQuery = sqlGenerator.addForUpdateClause(selectQuery);
new RetryingExecutor(hiveConf, () -> {
  Query query = pm.newQuery("javax.jdo.query.SQL", selectForUpdateQuery);
  query.setUnique(true);
  // only need to execute it to get db Lock
  query.execute();
}).run();
  }
{code}

The selectQuery is never closed.

> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>
> In ObjectStore.addNotificationEvent():
> {code}
>   Query objectQuery = pm.newQuery(MNotificationNextId.class);
>   Collection ids = (Collection) 
> objectQuery.execute();
> {code}
> The query is never closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov reassigned HIVE-17735:
-

Assignee: (was: Alexander Kolbasov)

> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>
> In ObjectStore.addNotificationEvent():
> {code}
>   Query objectQuery = pm.newQuery(MNotificationNextId.class);
>   Collection ids = (Collection) 
> objectQuery.execute();
> {code}
> The query is never closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17735) ObjectStore.addNotificationEvent is leaking queries

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov reassigned HIVE-17735:
-


> ObjectStore.addNotificationEvent is leaking queries
> ---
>
> Key: HIVE-17735
> URL: https://issues.apache.org/jira/browse/HIVE-17735
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> In ObjectStore.addNotificationEvent():
> {code}
>   Query objectQuery = pm.newQuery(MNotificationNextId.class);
>   Collection ids = (Collection) 
> objectQuery.execute();
> {code}
> The query is never closed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17718) HS2 Logs print unnecessary stack trace when HoS query is cancelled

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195448#comment-16195448
 ] 

Hive QA commented on HIVE-17718:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890772/HIVE-17718.3.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7166/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7166/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7166/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890772 - PreCommit-HIVE-Build

> HS2 Logs print unnecessary stack trace when HoS query is cancelled
> --
>
> Key: HIVE-17718
> URL: https://issues.apache.org/jira/browse/HIVE-17718
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17718.1.patch, HIVE-17718.2.patch, 
> HIVE-17718.3.patch
>
>
> Example:
> {code}
> 2017-10-05 17:47:11,881 ERROR 
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: 
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with 
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.status.RemoteSparkJobMonitor.startMonitor(RemoteSparkJobMonitor.java:124)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.status.impl.RemoteSparkJobRef.monitorJob(RemoteSparkJobRef.java:60)
>   at 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:111)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-10-05 17:47:11,881 WARN  org.apache.hadoop.hive.ql.Driver: 
> [HiveServer2-Handler-Pool: Thread-105]: Shutting down task : Stage-2:MAPRED
> 2017-10-05 17:47:11,882 ERROR 
> org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor: 
> [HiveServer2-Background-Pool: Thread-131]: Failed to monitor Job[ 2] with 
> exception 'java.lang.InterruptedException(sleep interrupted)'
> java.lang.InterruptedException: sleep interrupted
>   at java.lang.Thread.sleep(Native Method)
>   at 
> 

[jira] [Assigned] (HIVE-11266) count(*) wrong result based on table statistics for external tables

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-11266:
--

Assignee: Jesus Camacho Rodriguez

> count(*) wrong result based on table statistics for external tables
> ---
>
> Key: HIVE-11266
> URL: https://issues.apache.org/jira/browse/HIVE-11266
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Simone Battaglia
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
>
> Hive returns wrong count result on an external table with table statistics if 
> I change table data files.
> This is the scenario in details:
> 1) create external table my_table (...) location 'my_location';
> 2) analyze table my_table compute statistics;
> 3) change/add/delete one or more files in 'my_location' directory;
> 4) select count(\*) from my_table;
> In this case the count query doesn't generate a MR job and returns the result 
> based on table statistics. This result is wrong because is based on 
> statistics stored in the Hive metastore and doesn't take into account 
> modifications introduced on data files.
> Obviously setting "hive.compute.query.using.stats" to FALSE this problem 
> doesn't occur but the default value of this property is TRUE.
> I thinks that also this post on stackoverflow, that shows another type of bug 
> in case of multiple insert, is related to the one that I reported:
> http://stackoverflow.com/questions/24080276/wrong-result-for-count-in-hive-table



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17566) Create schema required for workload management.

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195411#comment-16195411
 ] 

Prasanth Jayachandran commented on HIVE-17566:
--

Created followup task to add a column for type of trigger. HIVE-17734

> Create schema required for workload management.
> ---
>
> Key: HIVE-17566
> URL: https://issues.apache.org/jira/browse/HIVE-17566
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17566.01.patch, HIVE-17566.02.patch, 
> HIVE-17566.03.patch, HIVE-17566.04.patch, HIVE-17566.05.patch, 
> HIVE-17566.06.patch, HIVE-17566.07.patch
>
>
> Schema + model changes required for workload management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17701) Added restriction to historic queries on web UI

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195400#comment-16195400
 ] 

Thejas M Nair commented on HIVE-17701:
--

+1


> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch, HIVE-17701.5.patch, HIVE-17701.6.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17300) WebUI query plan graphs

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195377#comment-16195377
 ] 

Hive QA commented on HIVE-17300:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890767/HIVE-17300.4.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=239)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDate (batchId=184)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimalX 
(batchId=184)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7165/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7165/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7165/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890767 - PreCommit-HIVE-Build

> WebUI query plan graphs
> ---
>
> Key: HIVE-17300
> URL: https://issues.apache.org/jira/browse/HIVE-17300
> Project: Hive
>  Issue Type: Improvement
>  Components: Web UI
>Reporter: Karen Coppage
>Assignee: Karen Coppage
> Attachments: complete_success.png, full_mapred_stats.png, 
> graph_with_mapred_stats.png, HIVE-17300.3.patch, HIVE-17300.4.patch, 
> HIVE-17300.patch, last_stage_error.png, last_stage_running.png, 
> non_mapred_task_selected.png
>
>
> Hi all,
> I’m working on a feature of the Hive WebUI Query Plan tab that would provide 
> the option to display the query plan as a nice graph (scroll down for 
> screenshots). If you click on one of the graph’s stages, the plan for that 
> stage appears as text below. 
> Stages are color-coded if they have a status (Success, Error, Running), and 
> the rest are grayed out. Coloring is based on status already available in the 
> WebUI, under the Stages tab.
> There is an additional option to display stats for MapReduce tasks. This 
> includes the job’s ID, tracking URL (where the logs are found), and mapper 
> and reducer numbers/progress, among other info. 
> The library I’m using for the graph is called vis.js (http://visjs.org/). It 
> has an Apache license, and the only necessary file to be included from this 
> library is about 700 KB.
> I tried to keep server-side changes minimal, and graph generation is taken 
> care of by the client. Plans with more than a given number of stages 
> (default: 25) won't be displayed in order to preserve resources.
> I’d love to hear any and all input from the community about this feature: do 
> you think it’s useful, and is there anything important I’m missing?
> Thanks,
> Karen Coppage
> Review request: https://reviews.apache.org/r/61663/
> Any input is welcome!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17721) with Postgres rdbms for metastore and dbnotification enabled, hive DDL SQL query fails

[Alexander Kolbasov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195373#comment-16195373
 ] 

Alexander Kolbasov commented on HIVE-17721:
---

[~spena] FYI

> with Postgres rdbms for metastore and dbnotification enabled, hive DDL SQL 
> query fails 
> ---
>
> Key: HIVE-17721
> URL: https://issues.apache.org/jira/browse/HIVE-17721
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
> Attachments: HIVE-17721.0.patch
>
>
> with postgres rdbms for hive-metastore any DDL fails when dbnotification is 
> enabled, the reason being a lock on the notification sequence is required, 
> which for Postgres requires the column-names and table names to enclosed in 
> "(double quotes) as we are using direct SQL and not going through datanucleus 
> and  postgres is case sensitive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement global execution triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.9.patch

Renamed Rules to Triggers all over

> Implement global execution triggers based on counters
> -
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.9.patch, HIVE-17508.WIP.2.patch, 
> HIVE-17508.WIP.patch
>
>
> Workload management can defined Triggers that are bound to a resource plan. 
> Each trigger can have a trigger expression and an action associated with it. 
> Trigger expressions are evaluated at runtime after configurable check 
> interval, based on which actions like killing a query, moving a query to 
> different pool etc. will get invoked. Simple execution trigger could be 
> something like
> {code}
> CREATE TRIGGER slow_query IN global
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17617) Rollup of an empty resultset should contain the grouping of the empty grouping set

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195337#comment-16195337
 ] 

Jesus Camacho Rodriguez commented on HIVE-17617:


[~kgyrtkirk], is this ready to be reviewed? Thanks

> Rollup of an empty resultset should contain the grouping of the empty 
> grouping set
> --
>
> Key: HIVE-17617
> URL: https://issues.apache.org/jira/browse/HIVE-17617
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-17617.01.patch, HIVE-17617.03.patch
>
>
> running
> {code}
> drop table if exists tx1;
> create table tx1 (a integer,b integer,c integer);
> select  sum(c),
> grouping(b)
> fromtx1
> group by rollup (b);
> {code}
> returns 0 rows; however 
> according to the standard:
> The  is regarded as the shortest such initial sublist. 
> For example, “ROLLUP ( (A, B), (C, D) )”
> is equivalent to “GROUPING SETS ( (A, B, C, D), (A, B), () )”.
> so I think the totals row (the grouping for {{()}} should be present)  - psql 
> returns it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement global execution triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Description: 
Workload management can defined Triggers that are bound to a resource plan. 
Each trigger can have a trigger expression and an action associated with it. 
Trigger expressions are evaluated at runtime after configurable check interval, 
based on which actions like killing a query, moving a query to different pool 
etc. will get invoked. Simple execution trigger could be something like
{code}
CREATE TRIGGER slow_query IN global
WHEN execution_time_ms > 1
MOVE TO slow_queue
{code}

  was:
Workload management can defined Rules that are bound to a resource plan. Each 
rule can have a trigger expression and an action associated with it. Trigger 
expressions are evaluated at runtime after configurable check interval, based 
on which actions like killing a query, moving a query to different pool etc. 
will get invoked. Simple rule could be something like
{code}
CREATE TRIGGER slow_query IN global
WHEN execution_time_ms > 1
MOVE TO slow_queue
{code}


> Implement global execution triggers based on counters
> -
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Triggers that are bound to a resource plan. 
> Each trigger can have a trigger expression and an action associated with it. 
> Trigger expressions are evaluated at runtime after configurable check 
> interval, based on which actions like killing a query, moving a query to 
> different pool etc. will get invoked. Simple execution trigger could be 
> something like
> {code}
> CREATE TRIGGER slow_query IN global
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement global execution triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Description: 
Workload management can defined Rules that are bound to a resource plan. Each 
rule can have a trigger expression and an action associated with it. Trigger 
expressions are evaluated at runtime after configurable check interval, based 
on which actions like killing a query, moving a query to different pool etc. 
will get invoked. Simple rule could be something like
{code}
CREATE TRIGGER slow_query IN global
WHEN execution_time_ms > 1
MOVE TO slow_queue
{code}

  was:
Workload management can defined Rules that are bound to a resource plan. Each 
rule can have a trigger expression and an action associated with it. Trigger 
expressions are evaluated at runtime after configurable check interval, based 
on which actions like killing a query, moving a query to different pool etc. 
will get invoked. Simple rule could be something like
{code}
CREATE TRIGGER slow_query IN resource_plan_name
WHEN execution_time_ms > 1
MOVE TO slow_queue
{code}


> Implement global execution triggers based on counters
> -
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE TRIGGER slow_query IN global
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement global execution triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Summary: Implement global execution triggers based on counters  (was: 
Implement pool rules and triggers based on counters)

> Implement global execution triggers based on counters
> -
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement global execution triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Description: 
Workload management can defined Rules that are bound to a resource plan. Each 
rule can have a trigger expression and an action associated with it. Trigger 
expressions are evaluated at runtime after configurable check interval, based 
on which actions like killing a query, moving a query to different pool etc. 
will get invoked. Simple rule could be something like
{code}
CREATE TRIGGER slow_query IN resource_plan_name
WHEN execution_time_ms > 1
MOVE TO slow_queue
{code}

  was:
Workload management can defined Rules that are bound to a resource plan. Each 
rule can have a trigger expression and an action associated with it. Trigger 
expressions are evaluated at runtime after configurable check interval, based 
on which actions like killing a query, moving a query to different pool etc. 
will get invoked. Simple rule could be something like
{code}
CREATE RULE slow_query IN resource_plan_name
WHEN execution_time_ms > 1
MOVE TO slow_queue
{code}


> Implement global execution triggers based on counters
> -
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE TRIGGER slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17702) incorrect isRepeating handling in decimal reader in ORC

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195331#comment-16195331
 ] 

Gopal V commented on HIVE-17702:


{code}
+  cv.noNulls = true;
+  cv.reset(); // Reset to work around some poor assumptions in ORC.
{code}

I see what you did there - LGTM - +1.

> incorrect isRepeating handling in decimal reader in ORC
> ---
>
> Key: HIVE-17702
> URL: https://issues.apache.org/jira/browse/HIVE-17702
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-17702.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17733) Move RawStore to standalone metastore

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195309#comment-16195309
 ] 

Sergey Shelukhin commented on HIVE-17733:
-

This patch will massively break HIVE-17629, that has been pending for a while. 
Is it possible to get that patch in before moving the class?

> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17733.patch
>
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17629) CachedStore - wait for prewarm at use time, not init time

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195311#comment-16195311
 ] 

Sergey Shelukhin commented on HIVE-17629:
-

[~thejas] [~vgumashta] [~daijy] ping? 

> CachedStore - wait for prewarm at use time, not init time
> -
>
> Key: HIVE-17629
> URL: https://issues.apache.org/jira/browse/HIVE-17629
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17629.patch
>
>
> Most of the changes are trivial, due to static changing to non-static. The 
> patch basically adds lifecycle management for shared cache, and uses it to 
> move prewarm to background thread, waiting for prewarm to happen at use time 
> not setConf, and to make init optional (if not called, CachedStore would 
> proxy the methods to rawStore, like it currently does in the ones not 
> implemented).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.8.patch

Added support for trigger based on TOTAL_TASKS. This will be required to avoid 
a task taking up too much of cluster resource by spinning up too many task. 

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.8.patch, HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17702) incorrect isRepeating handling in decimal reader in ORC

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17702:

Status: Patch Available  (was: Open)

> incorrect isRepeating handling in decimal reader in ORC
> ---
>
> Key: HIVE-17702
> URL: https://issues.apache.org/jira/browse/HIVE-17702
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-17702.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17702) incorrect isRepeating handling in decimal reader in ORC

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17702:

Attachment: HIVE-17702.patch

[~gopalv] can you take a look? thanks

> incorrect isRepeating handling in decimal reader in ORC
> ---
>
> Key: HIVE-17702
> URL: https://issues.apache.org/jira/browse/HIVE-17702
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-17702.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17733) Move RawStore to standalone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17733:
--
Status: Patch Available  (was: Open)

> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17733.patch
>
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17733) Move RawStore to standalone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17733:
--
Attachment: HIVE-17733.patch

> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17733.patch
>
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17714) move custom SerDe schema considerations into metastore from QL

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17714:

Description: 
Columns in metastore for tables that use external schema don't have the type 
information (since HIVE-11985) and may be entirely inconsistent (since forever, 
due to issues like HIVE-17713; or for SerDes that allow an URL for the schema, 
due to a change in the underlying file).
Currently, if you trace the usage of ConfVars.SERDESUSINGMETASTOREFORSCHEMA, 
and to MetaStoreUtils.getFieldsFromDeserializer, you'd see that the code in QL 
handles this in Hive. So, for the most part metastore just returns whatever is 
stored for columns in the database.

One exception appears to be get_fields_with_environment_context, which is 
interesting... so getTable will return incorrect columns (potentially), but 
get_fields/get_schema will return correct ones from SerDe as far as I can tell.

As part of separating the metastore, we should make sure all the APIs return 
the correct schema for the columns; it's not a good idea to have everyone 
reimplement getFieldsFromDeserializer.

Note: this should also remove a flag introduced in HIVE-17731


  was:
Columns in metastore for tables that use external schema don't have the type 
information (since HIVE-11985) and may be entirely inconsistent (since forever, 
due to issues like HIVE-17713; or for SerDes that allow an URL for the schema, 
due to a change in the underlying file).
Currently, if you trace the usage of ConfVars.SERDESUSINGMETASTOREFORSCHEMA, 
and to MetaStoreUtils.getFieldsFromDeserializer, you'd see that the code in QL 
handles this in Hive. So, for the most part metastore just returns whatever is 
stored for columns in the database.

One exception appears to be get_fields_with_environment_context, which is 
interesting... so getTable will return incorrect columns (potentially), but 
get_fields/get_schema will return correct ones from SerDe as far as I can tell.

As part of separating the metastore, we should make sure all the APIs return 
the correct schema for the columns; it's not a good idea to have everyone 
reimplement getFieldsFromDeserializer.





> move custom SerDe schema considerations into metastore from QL
> --
>
> Key: HIVE-17714
> URL: https://issues.apache.org/jira/browse/HIVE-17714
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Alan Gates
>
> Columns in metastore for tables that use external schema don't have the type 
> information (since HIVE-11985) and may be entirely inconsistent (since 
> forever, due to issues like HIVE-17713; or for SerDes that allow an URL for 
> the schema, due to a change in the underlying file).
> Currently, if you trace the usage of ConfVars.SERDESUSINGMETASTOREFORSCHEMA, 
> and to MetaStoreUtils.getFieldsFromDeserializer, you'd see that the code in 
> QL handles this in Hive. So, for the most part metastore just returns 
> whatever is stored for columns in the database.
> One exception appears to be get_fields_with_environment_context, which is 
> interesting... so getTable will return incorrect columns (potentially), but 
> get_fields/get_schema will return correct ones from SerDe as far as I can 
> tell.
> As part of separating the metastore, we should make sure all the APIs return 
> the correct schema for the columns; it's not a good idea to have everyone 
> reimplement getFieldsFromDeserializer.
> Note: this should also remove a flag introduced in HIVE-17731



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17733) Move RawStore to standalone metastore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195295#comment-16195295
 ] 

ASF GitHub Bot commented on HIVE-17733:
---

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/258

HIVE-17733

Moved RawStore, implementations, and some classes they depend on.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive17733

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/258.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #258


commit c6221d3305e0be74b63ec13902bff90cb355ccd1
Author: Alan Gates 
Date:   2017-08-24T00:04:18Z

Moved Filter.g and ExpressionTree.  As part of this I had to move 
PARTITION_DATE_FORMAT from HiveMetaStore to MetaStoreUtils.  Without this the 
move would have had to wait until I moved HiveMetaStore.  But this caused a 
cascade of other things to have to wait, so that I would have ended up with one 
huge patch that moves everything.

commit b7a555b27e993250ec2f7c53916d8aaf4d56e495
Author: Alan Gates 
Date:   2017-08-24T22:53:20Z

Moved NDV classes in prep for moving Stats mergers.

commit fa7602a290f5691242319cf74fbd021808cc088b
Author: Alan Gates 
Date:   2017-08-24T23:12:36Z

Moved ColumnStatsMerger and implementors.

commit 7089ee49ee579e0ef034d73543e96db207c871f1
Author: Alan Gates 
Date:   2017-08-25T00:08:34Z

Moved MetaStoreDirectSql, StatObjectConverter, and ColumnStatsAggregator 
and implementations.  Combined all of these together because there was a 
circular dependence.

commit e1d74d08237e7ddcdbb64a90915771ccd5d20199
Author: Alan Gates 
Date:   2017-08-25T23:37:33Z

Moved PartFilterExprUtil

commit fd6b77ee78dece36dd239510dc4c9623cd81370c
Author: Alan Gates 
Date:   2017-08-31T00:00:05Z

Moved stats data inspectors

commit 9fb4cede04a302f90641aec94a1f91c91a120b79
Author: Alan Gates 
Date:   2017-08-26T01:36:15Z

Moved RawStore and friends.

commit 416d88ae19c5c51bc97ba5c12d3dd03bc58e36f7
Author: Alan Gates 
Date:   2017-08-28T22:16:02Z

Moved RawStoreProxy and associated test.

commit 6aac3bb41b5923e179b7c508a0c0373e2c217f6f
Author: Alan Gates 
Date:   2017-10-06T21:51:23Z

Changes to make it compile after rebase.  Also split TestObjectStore so 
that I could move the majority of it to standalone-metastore (since ObjectStore 
is now there) but left a couple of tests in metastore since they reference 
EventMessage which hasn't moved yet.




> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17733) Move RawStore to standalone metastore

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17733:
--
Labels: pull-request-available  (was: )

> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17731) add a backward compat option for external users to HIVE-11985

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195292#comment-16195292
 ] 

Jesus Camacho Rodriguez commented on HIVE-17731:


LGTM, +1 (pending tests)

> add a backward compat option for external users to HIVE-11985
> -
>
> Key: HIVE-17731
> URL: https://issues.apache.org/jira/browse/HIVE-17731
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17731.patch
>
>
> See HIVE-11985.
> Some external callers (e.g. Presto) do not appear to process types from 
> deserializer correctly, relying on DB types. Ideally, it should be resolved 
> via HIVE-17714, hiding the custom SerDe logic from users.
> For now we can add a backward compatibility config for such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Attachment: HIVE-17701.6.patch

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch, HIVE-17701.5.patch, HIVE-17701.6.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17702) incorrect isRepeating handling in decimal reader in ORC

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17702:

Summary: incorrect isRepeating handling in decimal reader in ORC  (was: 
incorrect isRepeating handling in decimal reader)

> incorrect isRepeating handling in decimal reader in ORC
> ---
>
> Key: HIVE-17702
> URL: https://issues.apache.org/jira/browse/HIVE-17702
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17702) incorrect isRepeating handling in decimal reader in ORC

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17702:

Fix Version/s: (was: 1.3.0)

> incorrect isRepeating handling in decimal reader in ORC
> ---
>
> Key: HIVE-17702
> URL: https://issues.apache.org/jira/browse/HIVE-17702
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17733) Move RawStore to standalone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates reassigned HIVE-17733:
-


> Move RawStore to standalone metastore
> -
>
> Key: HIVE-17733
> URL: https://issues.apache.org/jira/browse/HIVE-17733
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
>
> This includes moving implementations of RawStore (like ObjectStore), 
> MetastoreDirectSql, and stats related classes like ColumnStatsAggregator and 
> the NDV classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17578) Create a TableRef object for Table/Partition

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195264#comment-16195264
 ] 

Hive QA commented on HIVE-17578:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890765/HIVE-17578.1.patch

{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=239)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7164/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7164/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7164/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890765 - PreCommit-HIVE-Build

> Create a TableRef object for Table/Partition
> 
>
> Key: HIVE-17578
> URL: https://issues.apache.org/jira/browse/HIVE-17578
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Gergely Hajós
> Attachments: HIVE-17578.1.patch
>
>
> a quick {{git grep DbName |grep -i TableName}} uncovers quite a lot of places 
> where the fully qualified {{dbname.tablename}} is being produced
> and most of the time the Table object is also present which might as well can 
> have a method to service a tableref.
> There might be some hidden bugs because of this...because at some places the 
> fully qualified table name is produced earlier...
> example callsite:
> https://github.com/apache/hive/blob/266c50554b86462d8b5ac84d28a2b237b8dbfa7e/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook.java#L63
> and called method:
> https://github.com/apache/hive/blob/266c50554b86462d8b5ac84d28a2b237b8dbfa7e/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L620



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17732) Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-17732:
---
Status: Patch Available  (was: Open)

> Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java
> -
>
> Key: HIVE-17732
> URL: https://issues.apache.org/jira/browse/HIVE-17732
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17732.patch
>
>
> Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe
> Remove superfluous logging, cut down on object instantiation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17732) Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-17732:
---
Attachment: HIVE-17732.patch

> Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java
> -
>
> Key: HIVE-17732
> URL: https://issues.apache.org/jira/browse/HIVE-17732
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17732.patch
>
>
> Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe
> Remove superfluous logging, cut down on object instantiation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17732) Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java

[BELUGA BEHR (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195251#comment-16195251
 ] 

BELUGA BEHR commented on HIVE-17732:


Just massaging the code into a better direction...

> Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java
> -
>
> Key: HIVE-17732
> URL: https://issues.apache.org/jira/browse/HIVE-17732
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: HIVE-17732.patch
>
>
> Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe
> Remove superfluous logging, cut down on object instantiation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17732) Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR updated HIVE-17732:
---
Summary: Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java  
(was: org.apache.hive.hcatalog.data.JsonSerDe.java)

> Minor Improvements - org.apache.hive.hcatalog.data.JsonSerDe.java
> -
>
> Key: HIVE-17732
> URL: https://issues.apache.org/jira/browse/HIVE-17732
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe
> Remove superfluous logging, cut down on object instantiation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17732) org.apache.hive.hcatalog.data.JsonSerDe.java

[BELUGA BEHR (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

BELUGA BEHR reassigned HIVE-17732:
--

Assignee: BELUGA BEHR

> org.apache.hive.hcatalog.data.JsonSerDe.java
> 
>
> Key: HIVE-17732
> URL: https://issues.apache.org/jira/browse/HIVE-17732
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
>
> Some simple improvements for org.apache.hive.hcatalog.data.JsonSerDe
> Remove superfluous logging, cut down on object instantiation 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17111) Add TestLocalSparkCliDriver

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195249#comment-16195249
 ] 

Sahil Takiar commented on HIVE-17111:
-

[~xuefuz]

{quote} 1. Other test driver class is generated, so I'm wondering why we 
actually have a nongenerated class for TestLocalSparkCliDriver. {quote} I think 
that was the old way of making these classes, now you can just directly create 
a new Java class to make a new CLI Driver. Unless I am misunderstanding the 
question.

{quote} 2. With this change, are we able to run any .q test using this test 
driver class? {quote}. Yes, the idea is that you can run any .q file with these 
new CLI Driver, which should help developers debug any HoS issues.

> Add TestLocalSparkCliDriver
> ---
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but 
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the 
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
>   // With local spark context, all user sessions share the same spark 
> context.
>   return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
>   return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and 
> sees that it doesn't start with {{local[}} and then decides to use the 
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed 
> up some of the tests, and also makes qtests easier to debug since everything 
> will now be run in the same process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17111) Add TestLocalSparkCliDriver

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195239#comment-16195239
 ] 

Xuefu Zhang commented on HIVE-17111:


I think this is good. However, I have two questions:

1. Other test driver class is generated, so I'm wondering why we actually have 
a nongenerated class for TestLocalSparkCliDriver.
2. With this change, are we able to run any .q test using this test driver 
class?

> Add TestLocalSparkCliDriver
> ---
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but 
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the 
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
>   // With local spark context, all user sessions share the same spark 
> context.
>   return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
>   return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and 
> sees that it doesn't start with {{local[}} and then decides to use the 
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed 
> up some of the tests, and also makes qtests easier to debug since everything 
> will now be run in the same process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17553) CBO wrongly type cast decimal literal to int

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-17553:
---
Status: Patch Available  (was: Open)

> CBO wrongly type cast decimal literal to int
> 
>
> Key: HIVE-17553
> URL: https://issues.apache.org/jira/browse/HIVE-17553
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17553.1.patch, HIVE-17553.2.patch
>
>
> {code:sql}explain select 100.000BD from f{code}
> {noformat}
> STAGE PLANS:
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
>   Processor Tree:
> TableScan
>   alias: f
>   Select Operator
> expressions: 100 (type: int)
> outputColumnNames: _col0
> ListSink
> {noformat}
> Notice that the expression 100.000BD is of type int instead of decimal.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17553) CBO wrongly type cast decimal literal to int

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-17553:
---
Attachment: HIVE-17553.2.patch

> CBO wrongly type cast decimal literal to int
> 
>
> Key: HIVE-17553
> URL: https://issues.apache.org/jira/browse/HIVE-17553
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17553.1.patch, HIVE-17553.2.patch
>
>
> {code:sql}explain select 100.000BD from f{code}
> {noformat}
> STAGE PLANS:
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
>   Processor Tree:
> TableScan
>   alias: f
>   Select Operator
> expressions: 100 (type: int)
> outputColumnNames: _col0
> ListSink
> {noformat}
> Notice that the expression 100.000BD is of type int instead of decimal.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17553) CBO wrongly type cast decimal literal to int

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-17553:
---
Status: Open  (was: Patch Available)

> CBO wrongly type cast decimal literal to int
> 
>
> Key: HIVE-17553
> URL: https://issues.apache.org/jira/browse/HIVE-17553
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17553.1.patch
>
>
> {code:sql}explain select 100.000BD from f{code}
> {noformat}
> STAGE PLANS:
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
>   Processor Tree:
> TableScan
>   alias: f
>   Select Operator
> expressions: 100 (type: int)
> outputColumnNames: _col0
> ListSink
> {noformat}
> Notice that the expression 100.000BD is of type int instead of decimal.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17731) add a backward compat option for external users to HIVE-11985

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17731:

Status: Patch Available  (was: Open)

> add a backward compat option for external users to HIVE-11985
> -
>
> Key: HIVE-17731
> URL: https://issues.apache.org/jira/browse/HIVE-17731
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17731.patch
>
>
> See HIVE-11985.
> Some external callers (e.g. Presto) do not appear to process types from 
> deserializer correctly, relying on DB types. Ideally, it should be resolved 
> via HIVE-17714, hiding the custom SerDe logic from users.
> For now we can add a backward compatibility config for such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17731) add a backward compat option for external users to HIVE-11985

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17731:

Attachment: HIVE-17731.patch

[~ashutoshc] would you like to take a look? A small patch.
The problem is that Presto tries to parse the column types instead of calling 
an equivalent of getTypesFromDeserializer. We could probably have better APIs 
for this (right now only getFields/getSchema returns the types from 
deserializer, getTable/etc. don't), I filed a separate JIRA for that.

> add a backward compat option for external users to HIVE-11985
> -
>
> Key: HIVE-17731
> URL: https://issues.apache.org/jira/browse/HIVE-17731
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-17731.patch
>
>
> See HIVE-11985.
> Some external callers (e.g. Presto) do not appear to process types from 
> deserializer correctly, relying on DB types. Ideally, it should be resolved 
> via HIVE-17714, hiding the custom SerDe logic from users.
> For now we can add a backward compatibility config for such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17731) add a backward compat option for external users to HIVE-11985

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-17731:
---


> add a backward compat option for external users to HIVE-11985
> -
>
> Key: HIVE-17731
> URL: https://issues.apache.org/jira/browse/HIVE-17731
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> See HIVE-11985.
> Some external callers (e.g. Presto) do not appear to process types from 
> deserializer correctly, relying on DB types. Ideally, it should be resolved 
> via HIVE-17714, hiding the custom SerDe logic from users.
> For now we can add a backward compatibility config for such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17725) Fix misnamed tests which are not run during precommit runs.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195188#comment-16195188
 ] 

Hive QA commented on HIVE-17725:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890758/HIVE-17725.1.patch

{color:green}SUCCESS:{color} +1 due to 29 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11212 tests 
executed
*Failed tests:*
{noformat}
TestHCatBase - did not produce a TEST-*.xml file (likely timed out) 
(batchId=190)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=232)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=240)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=240)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7163/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7163/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7163/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890758 - PreCommit-HIVE-Build

> Fix misnamed tests which are not run during precommit runs. 
> 
>
> Key: HIVE-17725
> URL: https://issues.apache.org/jira/browse/HIVE-17725
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Reporter: Zoltan Haindrich
>Assignee: Daniel Voros
> Attachments: HIVE-17725.1.patch
>
>
> I've just seen a testfailure for jdk9; but the test is not even executed 
> during precommit runs:
> {{TaskTrackerTest}}
> I think the test classes name should match the {{\*\*/Test\*}} to be executed 
> during test runs...however there seems to be quite a few... {{find . -name 
> '*Test.java'}} returns a few abstracts ; but there are real test ; which are 
> just misnamed...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17508) Implement pool rules and triggers based on counters

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-17508:
-
Attachment: HIVE-17508.7.patch

Fixes TestHiveConf test failure

> Implement pool rules and triggers based on counters
> ---
>
> Key: HIVE-17508
> URL: https://issues.apache.org/jira/browse/HIVE-17508
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-17508.1.patch, HIVE-17508.2.patch, 
> HIVE-17508.3.patch, HIVE-17508.3.patch, HIVE-17508.4.patch, 
> HIVE-17508.5.patch, HIVE-17508.6.patch, HIVE-17508.7.patch, 
> HIVE-17508.WIP.2.patch, HIVE-17508.WIP.patch
>
>
> Workload management can defined Rules that are bound to a resource plan. Each 
> rule can have a trigger expression and an action associated with it. Trigger 
> expressions are evaluated at runtime after configurable check interval, based 
> on which actions like killing a query, moving a query to different pool etc. 
> will get invoked. Simple rule could be something like
> {code}
> CREATE RULE slow_query IN resource_plan_name
> WHEN execution_time_ms > 1
> MOVE TO slow_queue
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17729) Add Database & Explain related blobstore tests

[Rentao Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rentao Wu updated HIVE-17729:
-
Status: Patch Available  (was: Open)

> Add Database & Explain related blobstore tests
> --
>
> Key: HIVE-17729
> URL: https://issues.apache.org/jira/browse/HIVE-17729
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Rentao Wu
>Assignee: Rentao Wu
> Attachments: HIVE-17729.patch
>
>
> This patch introduces the following regression tests into the hive-blobstore 
> qtest module:
> * create_database.q  -> tests tables with location inherited from database
> * multiple_db.q  -> tests query spanning multiple databases
> * explain.q -> tests EXPLAIN INSERT OVERWRITE command
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195075#comment-16195075
 ] 

Tao Li commented on HIVE-17701:
---

In the latest patch, we want to display sessions/queries that belong to the 
remote user if not an admin, when auth is enabled. When auth is disabled, each 
user acts as an admin and can view everything.

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch, HIVE-17701.5.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17729) Add Database & Explain related blobstore tests

[Rentao Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195072#comment-16195072
 ] 

Rentao Wu commented on HIVE-17729:
--

Attached patch and reviewboard cr

> Add Database & Explain related blobstore tests
> --
>
> Key: HIVE-17729
> URL: https://issues.apache.org/jira/browse/HIVE-17729
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Rentao Wu
>Assignee: Rentao Wu
> Attachments: HIVE-17729.patch
>
>
> This patch introduces the following regression tests into the hive-blobstore 
> qtest module:
> * create_database.q  -> tests tables with location inherited from database
> * multiple_db.q  -> tests query spanning multiple databases
> * explain.q -> tests EXPLAIN INSERT OVERWRITE command
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Attachment: HIVE-17701.5.patch

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch, HIVE-17701.5.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17729) Add Database & Explain related blobstore tests

[Rentao Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rentao Wu updated HIVE-17729:
-
Attachment: HIVE-17729.patch

> Add Database & Explain related blobstore tests
> --
>
> Key: HIVE-17729
> URL: https://issues.apache.org/jira/browse/HIVE-17729
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Rentao Wu
>Assignee: Rentao Wu
> Attachments: HIVE-17729.patch
>
>
> This patch introduces the following regression tests into the hive-blobstore 
> qtest module:
> * create_database.q  -> tests tables with location inherited from database
> * multiple_db.q  -> tests query spanning multiple databases
> * explain.q -> tests EXPLAIN INSERT OVERWRITE command
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17730) Queries can be closed automatically

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov reassigned HIVE-17730:
-

Assignee: Alexander Kolbasov

> Queries can be closed automatically
> ---
>
> Key: HIVE-17730
> URL: https://issues.apache.org/jira/browse/HIVE-17730
> Project: Hive
>  Issue Type: Bug
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>
> HIVE-16213 made QueryWrapper AutoCloseable, but queries are still closed 
> manually and not by using try-with-resource. And now Query itself is auto 
> closeable, so we don't need the wrapper at all.
> So we should get rid of QueryWrapper and use try-with-resource to create 
> queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17566) Create schema required for workload management.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195042#comment-16195042
 ] 

Hive QA commented on HIVE-17566:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890736/HIVE-17566.07.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=231)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=231)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=239)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout 
(batchId=228)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7162/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7162/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7162/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890736 - PreCommit-HIVE-Build

> Create schema required for workload management.
> ---
>
> Key: HIVE-17566
> URL: https://issues.apache.org/jira/browse/HIVE-17566
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Harish Jaiprakash
>Assignee: Harish Jaiprakash
> Attachments: HIVE-17566.01.patch, HIVE-17566.02.patch, 
> HIVE-17566.03.patch, HIVE-17566.04.patch, HIVE-17566.05.patch, 
> HIVE-17566.06.patch, HIVE-17566.07.patch
>
>
> Schema + model changes required for workload management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17111) Add TestLocalSparkCliDriver

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195041#comment-16195041
 ] 

Aihua Xu commented on HIVE-17111:
-

TestSparkCliDriver is testing on local cluster. TestMiniSparkOnYarnCliDriver is 
testing against yarn. I see the value to add TestLocalSparkCliDriver to cover 
LocalHiveSparkClient and also it will make debugging Spark  easier. Of course 
we just need one or two tests to cover it as the patch does. 

+1 from me.

[~lirui], [~xuefuz] How do you think?
 

> Add TestLocalSparkCliDriver
> ---
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but 
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the 
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
>   // With local spark context, all user sessions share the same spark 
> context.
>   return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
>   return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and 
> sees that it doesn't start with {{local[}} and then decides to use the 
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed 
> up some of the tests, and also makes qtests easier to debug since everything 
> will now be run in the same process.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195002#comment-16195002
 ] 

Vihang Karajgaonkar commented on HIVE-14389:


Thats the behavior which was fixed by this patch. The query text should not be 
mixed with the results because if you want to process the results (to load as 
data somewhere else for example)  then you will have to manually clean it up by 
removing the query text. You can always add the query text to your results file 
if really want by redirecting the stderr to the same file using {{beeline 
--verbose=true -f hadoop_query.hql > test.txt 2>&1}}

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch, 
> HIVE-14389.03.patch, HIVE-14389.04.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17371) Move tokenstores to metastore module

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17371:
---
Attachment: HIVE-17371.03.patch

> Move tokenstores to metastore module
> 
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch, HIVE-17371.02.patch, 
> HIVE-17371.03.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and 
> {{ZKTokenStore}} since they implement 
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of  
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws 
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf, 
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing 
> in a HiveConf for
> // configuration.  It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its 
> construction.  But this is
> // the hive-shims version of the memory store.  We want to convert this 
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
> 
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName)) 
> {
>   return new MemoryTokenStore();
> }
> try {
>   Class storeClass =
>   
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
>   return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
>   throw new IOException("Error initializing delegation token store: " + 
> tokenStoreClassName, e);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17371) Move tokenstores to metastore module

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17371:
---
Attachment: (was: HIVE-17371.03.patch)

> Move tokenstores to metastore module
> 
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch, HIVE-17371.02.patch, 
> HIVE-17371.03.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and 
> {{ZKTokenStore}} since they implement 
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of  
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws 
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf, 
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing 
> in a HiveConf for
> // configuration.  It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its 
> construction.  But this is
> // the hive-shims version of the memory store.  We want to convert this 
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
> 
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName)) 
> {
>   return new MemoryTokenStore();
> }
> try {
>   Class storeClass =
>   
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
>   return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
>   throw new IOException("Error initializing delegation token store: " + 
> tokenStoreClassName, e);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17371) Move tokenstores to metastore module

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17371:
---
Attachment: HIVE-17371.03.patch

Rebased to latest code and resolved conflicts.

> Move tokenstores to metastore module
> 
>
> Key: HIVE-17371
> URL: https://issues.apache.org/jira/browse/HIVE-17371
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17371.01.patch, HIVE-17371.02.patch, 
> HIVE-17371.03.patch
>
>
> The {{getTokenStore}} method will not work for the {{DBTokenStore}} and 
> {{ZKTokenStore}} since they implement 
> {{org.apache.hadoop.hive.thrift.DelegationTokenStore}} instead of  
> {{org.apache.hadoop.hive.metastore.security.DelegationTokenStore}}
> {code}
> private DelegationTokenStore getTokenStore(Configuration conf) throws 
> IOException {
> String tokenStoreClassName =
> MetastoreConf.getVar(conf, 
> MetastoreConf.ConfVars.DELEGATION_TOKEN_STORE_CLS, "");
> // The second half of this if is to catch cases where users are passing 
> in a HiveConf for
> // configuration.  It will have set the default value of
> // "hive.cluster.delegation.token.store .class" to
> // "org.apache.hadoop.hive.thrift.MemoryTokenStore" as part of its 
> construction.  But this is
> // the hive-shims version of the memory store.  We want to convert this 
> to our default value.
> if (StringUtils.isBlank(tokenStoreClassName) ||
> 
> "org.apache.hadoop.hive.thrift.MemoryTokenStore".equals(tokenStoreClassName)) 
> {
>   return new MemoryTokenStore();
> }
> try {
>   Class storeClass =
>   
> Class.forName(tokenStoreClassName).asSubclass(DelegationTokenStore.class);
>   return ReflectionUtils.newInstance(storeClass, conf);
> } catch (ClassNotFoundException e) {
>   throw new IOException("Error initializing delegation token store: " + 
> tokenStoreClassName, e);
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17729) Add Database & Explain related blobstore tests

[Rentao Wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rentao Wu reassigned HIVE-17729:



> Add Database & Explain related blobstore tests
> --
>
> Key: HIVE-17729
> URL: https://issues.apache.org/jira/browse/HIVE-17729
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Rentao Wu
>Assignee: Rentao Wu
>
> This patch introduces the following regression tests into the hive-blobstore 
> qtest module:
> * create_database.q  -> tests tables with location inherited from database
> * multiple_db.q  -> tests query spanning multiple databases
> * explain.q -> tests EXPLAIN INSERT OVERWRITE command
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Sanman Ganthe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194981#comment-16194981
 ] 

Sanman Ganthe commented on HIVE-14389:
--

Command:
beeline --verbose=true -f hadoop_query.hql > test.txt

Output in test.txt
+---+-+-
| column 1  | column 2|
+---+-+-
| A | AA  |
| A | AA  |
| A | AA  |
| A | AA  |

I want to see the query and output. This is what we would get before HDP 2.6.2
 
+---+-+-
| column 1  | column 2|
+---+-+-
| A | AA  |
| A | AA  |
| A | AA  |
| A | AA  |

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch, 
> HIVE-14389.03.patch, HIVE-14389.04.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17720) Bitvectors are not shown in describe statement on beeline

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194950#comment-16194950
 ] 

Hive QA commented on HIVE-17720:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890734/HIVE-17720.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11191 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
 (batchId=232)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
 (batchId=232)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
 (batchId=243)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_5] 
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9] 
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl] 
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_partlvl_dp] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnstats_tbllvl] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compute_stats_date] 
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[display_colstats_tbllvl] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fm-sketch] (batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[hll] (batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partial_column_stats] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_display_colstats_tbllvl]
 (batchId=75)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] 
(batchId=101)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=240)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=240)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7161/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7161/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7161/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890734 - PreCommit-HIVE-Build

> Bitvectors are not shown in describe statement on beeline
> -
>
> Key: HIVE-17720
> URL: https://issues.apache.org/jira/browse/HIVE-17720
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Diagnosability
>Affects Versions: 3.0.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-17720.02.patch, HIVE-17720.patch
>
>
> Describe statement takes different codepath for HS2 where bit vectors weren't 
> displayed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194899#comment-16194899
 ] 

Vihang Karajgaonkar commented on HIVE-14389:


Hi [~sanmanganthe] Can you please provide more details with an example? Do you 
mean that the query is not available on stderr as well?

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch, 
> HIVE-14389.03.patch, HIVE-14389.04.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Status: Patch Available  (was: Open)

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Attachment: HIVE-17701.4.patch

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17572) Warnings from SparkCrossProductCheck for MapJoins are confusing

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman reassigned HIVE-17572:
-

Assignee: Andrew Sherman

> Warnings from SparkCrossProductCheck for MapJoins are confusing
> ---
>
> Key: HIVE-17572
> URL: https://issues.apache.org/jira/browse/HIVE-17572
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
>
> When the {{SparkCrossProductCheck}} detects a cross-product in a map-join, it 
> prints out a confusing warning - e.g. {{Map Join MAPJOIN\[9\]\[bigTable=?\] 
> in task 'Stage-1:MAPRED' is a cross product}}
> I see a few ways this can be imrpoved:
> * {{bigTable}} should actually specify the big table
> * I'm not sure why the stage id is printed instead of the work id, when a 
> cross product is detected in a shuffle join the work id is shown (e.g. 
> {{Warning: Shuffle Join JOIN\[13\]\[tables = \[$hdt$_1, $hdt$_2, $hdt$_0\]\] 
> in Work 'Reducer 3' is a cross product}})
> * It shouldn't say {{MAPRED}} that can be confusing to users
> * The {{MAPJOIN}} id doesn't need to be printed, it doesn't have any meaning 
> to the user and the value just keeps on going up and up the longer a session 
> lives
> On a somewhat related note, could we just stick this warning in the explain 
> plan? Otherwise users may not even notice it



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Status: Open  (was: Patch Available)

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Attachment: HIVE-17701.4.patch

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch, HIVE-17701.4.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17701) Added restriction to historic queries on web UI

[Tao Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Li updated HIVE-17701:
--
Attachment: (was: HIVE-17701.4.patch)

> Added restriction to historic queries on web UI
> ---
>
> Key: HIVE-17701
> URL: https://issues.apache.org/jira/browse/HIVE-17701
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Tao Li
> Attachments: HIVE-17701.1.patch, HIVE-17701.2.patch, 
> HIVE-17701.3.patch
>
>
> The HiveServer2 Web UI (HIVE-12550) shows recently completed queries. 
> However, a user can see the queries run by other users as well, and that is a 
> security/privacy concern.
> Only admin users should be allowed to see queries from other users (similar 
> to behavior of display for configs, stack trace etc).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17727) HoS Queries Print "Starting task [Stage-x:MAPRED] in serial mode"

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman reassigned HIVE-17727:
-

Assignee: Andrew Sherman

> HoS Queries Print "Starting task [Stage-x:MAPRED] in serial mode"
> -
>
> Key: HIVE-17727
> URL: https://issues.apache.org/jira/browse/HIVE-17727
> Project: Hive
>  Issue Type: Improvement
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
>
> Whenever a HoS query is run something like "Starting task [Stage-3:MAPRED] in 
> serial mode" in printed out for each {{SparkTask}}, which is confusing 
> because this isn't a MAPRED job. We should change {{StageType}} to include a 
> {{SPARK}} type.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17561) Move TxnStore and implementations to standalone metastore

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-17561:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Patch 6 committed.  Thanks Eugene for the reviews.

> Move TxnStore and implementations to standalone metastore
> -
>
> Key: HIVE-17561
> URL: https://issues.apache.org/jira/browse/HIVE-17561
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore, Transactions
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17561.4.patch, HIVE-17561.5.patch, 
> HIVE-17561.6.patch, HIVE-17561.patch
>
>
> We need to move the metastore handling of transactions into the standalone 
> metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17723) Update Accumulo drive q.out files

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194843#comment-16194843
 ] 

Aihua Xu commented on HIVE-17723:
-

Thanks [~pvary] to take care of that. +1.

> Update Accumulo drive q.out files
> -
>
> Key: HIVE-17723
> URL: https://issues.apache.org/jira/browse/HIVE-17723
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17723.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17691) Miscellaneous List

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17691:
--
Summary: Miscellaneous List  (was: Misc Nits)

> Miscellaneous List
> --
>
> Key: HIVE-17691
> URL: https://issues.apache.org/jira/browse/HIVE-17691
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>
> # DDLSemanticAnalyzer.alterTableOutput is unused
> # DDLTask.generateAddMmTasks(Table) - stmtId should probably come from 
> TransactionManager
> # DDLTask.createTable(Hive db, CreateTableDesc crtTbl) has _Long mmWriteId = 
> crtTbl.getInitialMmWriteId();_ logic is unclear..  this ID is only set in one 
> place..
> # FileSinkOperator has multiple places that look like _conf.getWriteType() == 
> AcidUtils.Operation.NOT_ACID || conf.isMmTable()_ - what is the writeType for 
> MM tables?  Seems that Wei opted for "work.getLoadTableWork().getWriteType() 
> != AcidUtils.Operation.NOT_ACID && !tbd.isMmTable()" to mean MM, e.g. 
> MoveTask.handleStaticParts() call to Hive.loadPartition()
> # HiveConf.HIVE_TXN_OPERATIONAL_PROPERTIES - the doc/explanation there is 
> obsolete
> # Compactor Initiator likely doesn't work for MM tables.  It's triggered by 
> into in TXN_COMPONENTS/COMPLETED_TXN_COMPONENTS.  MM tables don't write to 
> either because DbTxnManager.acquireLocks() does  
> _compBuilder.setIsAcid(AcidUtils.isFullAcidTable(t));_ i.e. it treats MM as 
> non-acid tables
> # In general integration with full Acid seems confused wrt to MM and seems to 
> treat MM as special table type rather than subtype of Acid table.  (mostly, 
> but not always).
> # LoadSemanticAnalyzer.analyzeInternal(ASTNode) sets statementId to 0 rather 
> than from TM
> # ImportCommitTask - doesn't currently do anything.  It used to commit mmID.  
> Need to verify we properly commit the txn in the Driver
> # As far as I can tell all the mm_*.q tests run on TestCliDriver which means 
> MR.  This doesn't exercise some code specifically for dealing with writes 
> from Union All queries (CTAS, Insert into).  On MR this requires 
> "hive.optimize.union.remove=true" (false by default)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194838#comment-16194838
 ] 

Thejas M Nair commented on HIVE-17728:
--

+1


> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17723) Update Accumulo drive q.out files

[Peter Vary (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194840#comment-16194840
 ] 

Peter Vary commented on HIVE-17723:
---

[~aihuaxu]: Could you review please?
Thanks,
Peter

> Update Accumulo drive q.out files
> -
>
> Key: HIVE-17723
> URL: https://issues.apache.org/jira/browse/HIVE-17723
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17723.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17723) Update Accumulo drive q.out files

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194820#comment-16194820
 ] 

Hive QA commented on HIVE-17723:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12890718/HIVE-17723.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11190 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
 (batchId=171)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14] 
(batchId=240)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23] 
(batchId=240)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7160/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7160/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7160/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12890718 - PreCommit-HIVE-Build

> Update Accumulo drive q.out files
> -
>
> Key: HIVE-17723
> URL: https://issues.apache.org/jira/browse/HIVE-17723
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17723.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-17728:

Status: Patch Available  (was: Open)

> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-17728:

Attachment: HIVE-17728.01.patch

Added 01.patch with changes in TestHCatClient to use transactional event 
listener.
Request [~thejas] to please review the same.

> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17728.01.patch
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work stopped] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-17728 stopped by Sankar Hariappan.
---
> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17728:
--
Labels: DR pull-request-available replication  (was: DR replication)

> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17728) TestHCatClient should configure hive.metastore.transactional.event.listeners as per recommendation.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194810#comment-16194810
 ] 

ASF GitHub Bot commented on HIVE-17728:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/257

HIVE-17728: TestHCatClient should configure 
hive.metastore.transactional.event.listeners as per recommendation.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-17728

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/257.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #257


commit e5be3c09e18398824b74c7294f1d5af881a83637
Author: Sankar Hariappan 
Date:   2017-10-06T16:31:43Z

HIVE-17728: TestHCatClient should configure 
hive.metastore.transactional.event.listeners as per recommendation.




> TestHCatClient should configure hive.metastore.transactional.event.listeners 
> as per recommendation.
> ---
>
> Key: HIVE-17728
> URL: https://issues.apache.org/jira/browse/HIVE-17728
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>  Labels: DR, pull-request-available, replication
> Fix For: 3.0.0
>
>
> Currently, TestHCatClient.java uses hive.metastore.event.listeners to enable 
> notification events logging. But the recommended configuration for the same 
> is hive.metastore.transactional.event.listeners. 
> So, need to update the same.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14389) Beeline should not output query and prompt to stdout

[Sanman Ganthe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194808#comment-16194808
 ] 

Sanman Ganthe commented on HIVE-14389:
--

Hi. I was not getting the query even after i used --verbose=true. I have to 
redirect the output with the query into a text file.

> Beeline should not output query and prompt to stdout
> 
>
> Key: HIVE-14389
> URL: https://issues.apache.org/jira/browse/HIVE-14389
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-14389.01.patch, HIVE-14389.02.patch, 
> HIVE-14389.03.patch, HIVE-14389.04.patch
>
>
> It seems that the Beeline prints the query along with the results in the 
> stdout when a script file is passed. The output file in the example below 
> needs to only have the results and not the query.
> {noformat}
> .vihang-MBP:bin vihang$ ./beeline --showheader=false --outformat=tsv2 -u 
> "jdbc:hive2://localhost:1" -f /tmp/query.sql > /tmp/query.out 2> 
> /tmp/query.err
> OK
> $ cat /tmp/query.out
> 1: jdbc:hive2://localhost:1/default> select * from likes limit 4;
> +---+--+--+
> | 1 | chocolate|
> | 1 | car  |
> | 1 | games|
> | 1 | chess|
> +---+--+--+
> 1: jdbc:hive2://localhost:1/default>
> 1: jdbc:hive2://localhost:1/default>
> $
> {noformat}
> A lot of people use HiveCLI and in order to transition from HiveCLI scripts 
> to Beeline, this needs to be taken care of. The output files generated by 
> beeline should contain only the results and nothing else.
> Similarly, when not in silent mode, query are being printed out on stdout, 
> which is adding garbage along with results, as just like HIVE CLI does, users 
> would like to have only the results on stdout, not errors/debugging info/etc, 
> like the full query. 
> Query could be printed out, no problem, as long as it is not on stdout (with 
> results), instead, it must be printed out along with the debugging info.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17561) Move TxnStore and implementations to standalone metastore

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194799#comment-16194799
 ] 

Eugene Koifman commented on HIVE-17561:
---

+1 patch 6

> Move TxnStore and implementations to standalone metastore
> -
>
> Key: HIVE-17561
> URL: https://issues.apache.org/jira/browse/HIVE-17561
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore, Transactions
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: pull-request-available
> Attachments: HIVE-17561.4.patch, HIVE-17561.5.patch, 
> HIVE-17561.6.patch, HIVE-17561.patch
>
>
> We need to move the metastore handling of transactions into the standalone 
> metastore.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)