[jira] [Commented] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292751#comment-15292751
 ] 

Lefty Leverenz commented on HIVE-13343:
---

Doc note:  This adds *hive.llap.enable.grace.join.in.llap* to HiveConf.java so 
it will need to be documented in the LLAP section of Configuration Properties 
for release 2.1.0.

* [Configuration Properties -- LLAP | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-LLAP]

A release note would be helpful too, and Fix Version needs to be set.

Should usage notes be documented in the LLAP wikidoc?

> Need to disable hybrid grace hash join in llap mode except for dynamically 
> partitioned hash join
> 
>
> Key: HIVE-13343
> URL: https://issues.apache.org/jira/browse/HIVE-13343
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>  Labels: TODOC2.1
> Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, 
> HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, 
> HIVE-13343.6.patch, HIVE-13343.7.patch
>
>
> Due to performance reasons, we should disable use of hybrid grace hash join 
> in llap when dynamic partition hash join is not used. With dynamic partition 
> hash join, we need hybrid grace hash join due to the possibility of skews.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13343) Need to disable hybrid grace hash join in llap mode except for dynamically partitioned hash join

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-13343:
--
Labels: TODOC2.1  (was: )

> Need to disable hybrid grace hash join in llap mode except for dynamically 
> partitioned hash join
> 
>
> Key: HIVE-13343
> URL: https://issues.apache.org/jira/browse/HIVE-13343
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>  Labels: TODOC2.1
> Attachments: HIVE-13343.1.patch, HIVE-13343.2.patch, 
> HIVE-13343.3.patch, HIVE-13343.4.patch, HIVE-13343.5.patch, 
> HIVE-13343.6.patch, HIVE-13343.7.patch
>
>
> Due to performance reasons, we should disable use of hybrid grace hash join 
> in llap when dynamic partition hash join is not used. With dynamic partition 
> hash join, we need hybrid grace hash join due to the possibility of skews.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Attachment: HIVE-13799.4.patch

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch, 
> HIVE-13799.3.patch, HIVE-13799.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Attachment: (was: HIVE-13799.4.patch)

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch, 
> HIVE-13799.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Attachment: HIVE-13799.4.patch

Thanks [~gopalv]. Attaching the revised patch.

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch, 
> HIVE-13799.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13782) Compile async query asynchronously

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292737#comment-15292737
 ] 

Lefty Leverenz commented on HIVE-13782:
---

Nitpicky edits for the parameter description:

{code}
+
HIVE_SERVER2_ASYNC_EXEC_ASYNC_COMPILE("hive.server2.async.exec.async.compile", 
false,
+"Whether enable compiling async query asynchronously. If enabled, 
result information is not available before compilation"),
{code}

That should be "Whether to enable ..." and the second sentence needs a period 
at the end.

By the way, "result information is not available before compilation" bewilders 
me because I assume results would always come last.  Does "result information" 
mean the result of compilation or the query results?  Does "before compilation" 
mean before compilation is completed?  (Please forgive my ignorance.)

> Compile async query asynchronously
> --
>
> Key: HIVE-13782
> URL: https://issues.apache.org/jira/browse/HIVE-13782
> Project: Hive
>  Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-13782.1.patch
>
>
> Currently, when an async query is submitted to HS2, HS2 does the preparation 
> synchronously. One of the preparation step is to compile the query, which may 
> take some time. It will be helpful to provide an option to do the compilation 
> asynchronously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13801) LLAP UI does not seem to accept credentials

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292705#comment-15292705
 ] 

Gopal V commented on HIVE-13801:


[~sseth]: any logs from the LLAP side - the auto-auth is configured exactly the 
same way the HS2 web UI is configured.

> LLAP UI does not seem to accept credentials 
> 
>
> Key: HIVE-13801
> URL: https://issues.apache.org/jira/browse/HIVE-13801
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Priority: Critical
>
> Effectively making it unusable on a secure cluster.
> This could well be a misconfiguration of the cluster - but I tried using the 
> same credentials against the YARN Timeline Server - and that worked fine.
> Steps to obtain credentials. kinit - start and configure firefox to use 
> SPNEGO, try accessing the UGI.
> cc [~gopalv]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292703#comment-15292703
 ] 

Gopal V commented on HIVE-13799:


Thanks [~rajesh.balamohan] for the update - the check can be higher up in the 
pipeline, the list of partitions are irrelevant if the table is not bucketed at 
all.

The Table Storage desc should have the number of buckets declared, similar to 
the flat table.

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch, 
> HIVE-13799.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Attachment: HIVE-13799.3.patch

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch, 
> HIVE-13799.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13249) Hard upper bound on number of open transactions

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-13249:
-
Attachment: HIVE-13249.12.patch

Patch 12, rebased.

> Hard upper bound on number of open transactions
> ---
>
> Key: HIVE-13249
> URL: https://issues.apache.org/jira/browse/HIVE-13249
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, 
> HIVE-13249.11.patch, HIVE-13249.12.patch, HIVE-13249.2.patch, 
> HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, 
> HIVE-13249.6.patch, HIVE-13249.7.patch, HIVE-13249.8.patch, HIVE-13249.9.patch
>
>
> We need to have a safeguard by adding an upper bound for open transactions to 
> avoid huge number of open-transaction requests, usually due to improper 
> configuration of clients such as Storm.
> Once that limit is reached, clients will start failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292671#comment-15292671
 ] 

Gopal V commented on HIVE-13799:


[~rajesh.balamohan]: if possible, please replace the variables with "final" if 
appropriate - gets easier to reason about code when most variables are 
immutable in scope.

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Attachment: HIVE-13799.2.patch

Attaching .2 version for review which handles both partitioned and 
non-partitioned tables.

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch, HIVE-13799.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Comment: was deleted

(was: For regular tables with no-buckets, it was making fs calls which is 
removed with the patch.)

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13783) No secondary prompt

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292609#comment-15292609
 ] 

Hive QA commented on HIVE-13783:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804791/HIVE-13783.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 10007 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketmapjoin3.q-enforce_order.q-union11.q-and-12-more - did 
not produce a TEST-*.xml file
TestSparkCliDriver-groupby_grouping_id2.q-vectorization_13.q-auto_sortmerge_join_13.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-join9.q-join_casesensitive.q-filter_join_breaktask.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-escape_distributeby1.q-auto_sortmerge_join_7.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-union_remove_23.q-transform_ppr2.q-join20.q-and-12-more - 
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.TestLlapDaemonProtocolServerImpl.test
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers
org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/331/testReport
Console output: 

[jira] [Updated] (HIVE-13800) Disable auth enabled by default on LLAP UI for secure clusters

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-13800:
--
Attachment: HIVE-13800.01.patch

Trivial patch. cc [~sershe], [~gopalv]

> Disable auth enabled by default on LLAP UI for secure clusters
> --
>
> Key: HIVE-13800
> URL: https://issues.apache.org/jira/browse/HIVE-13800
> Project: Hive
>  Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-13800.01.patch
>
>
> There's no sensitive information that I'm aware of. (The logs would be the 
> most sensitive).
> Similar to the HS2 UI, the LLAP UI can be default unprotected even on secure 
> clusters.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13799) Optimize TableScanRule::checkBucketedTable

[Rajesh Balamohan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan updated HIVE-13799:

Attachment: HIVE-13799.1.patch

For regular tables with no-buckets, it was making fs calls which is removed 
with the patch.

> Optimize TableScanRule::checkBucketedTable
> --
>
> Key: HIVE-13799
> URL: https://issues.apache.org/jira/browse/HIVE-13799
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-13799.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13391:

Attachment: HIVE-13391.09.patch

Rebased the patch. I will file a follow-up JIRA to consider the approach from 
02 patch instead, to avoid running UDFs under kerberos superuser. But most of 
the code will remain the same...

> add an option to LLAP to use keytab to authenticate to read data
> 
>
> Key: HIVE-13391
> URL: https://issues.apache.org/jira/browse/HIVE-13391
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch, 
> HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch, 
> HIVE-13391.06.patch, HIVE-13391.07.patch, HIVE-13391.08.patch, 
> HIVE-13391.09.patch, HIVE-13391.patch
>
>
> This can be used for non-doAs case to allow access to clients who don't 
> propagate HDFS tokens.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13443) LLAP: signing for the second state of submit (the event)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292482#comment-15292482
 ] 

Hive QA commented on HIVE-13443:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804821/HIVE-13443.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/330/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/330/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-330/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[INFO] Building Hive Integration - Testing Utilities 2.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-util ---
[INFO] Deleting 
/data/hive-ptest/working/apache-github-source-source/itests/util/target
[INFO] Deleting 
/data/hive-ptest/working/apache-github-source-source/itests/util (includes = 
[datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ 
hive-it-util ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-util ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-it-util 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it-util ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-it-util 
---
[INFO] Compiling 52 source files to 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/classes
[WARNING] 
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java:
 Some input files use or override a deprecated API.
[WARNING] 
/data/hive-ptest/working/apache-github-source-source/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java:
 Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ 
hive-it-util ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-github-source-source/itests/util/src/test/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-util ---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/warehouse
[mkdir] Created dir: 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
 [copy] Copying 15 files to 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hive-it-util ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-it-util ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-util ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-2.1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ 
hive-it-util ---
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it-util ---
[INFO] Installing 
/data/hive-ptest/working/apache-github-source-source/itests/util/target/hive-it-util-2.1.0-SNAPSHOT.jar
 to 
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/2.1.0-SNAPSHOT/hive-it-util-2.1.0-SNAPSHOT.jar
[INFO] Installing 
/data/hive-ptest/working/apache-github-source-source/itests/util/pom.xml to 
/home/hiveptest/.m2/repository/org/apache/hive/hive-it-util/2.1.0-SNAPSHOT/hive-it-util-2.1.0-SNAPSHOT.pom
[INFO] 
[INFO] 
[INFO] Building Hive Integration - Unit Tests 2.1.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-unit ---

[jira] [Commented] (HIVE-13782) Compile async query asynchronously

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292474#comment-15292474
 ] 

Hive QA commented on HIVE-13782:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804787/HIVE-13782.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 9250 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-join2.q-input17.q-and-12-more - did not 
produce a TEST-*.xml file
TestSparkCliDriver-groupby_grouping_id2.q-vectorization_13.q-auto_sortmerge_join_13.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-join_cond_pushdown_unqual4.q-bucketmapjoin12.q-avro_decimal_native.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/329/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/329/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-329/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12804787 - PreCommit-HIVE-MASTER-Build

> Compile async query asynchronously
> --
>
> 

[jira] [Commented] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292441#comment-15292441
 ] 

Ashutosh Chauhan commented on HIVE-13798:
-

I think your change is more portable (across different OS) so its a useful 
change. +1
Just for sanity lets wait for QA run.

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292417#comment-15292417
 ] 

Aihua Xu commented on HIVE-13798:
-

[~ashutoshc] Another test case failure with the following exception. I'm not 
sure if changing conf/ivysettings.xml would affect other tests but seems it's 
added for this test case.

{noformat}
2016-05-18T17:01:32,045 ERROR [0bf184af-bfd0-4e84-b3dd-865205011889 main[]]: 
SessionState (SessionState.java:printError(1060)) - Error grabbing Grapes -- 
[unresolved dependency: #udfexampleadd;1.0: not found]
java.lang.RuntimeException: Error grabbing Grapes -- [unresolved dependency: 
#udfexampleadd;1.0: not found]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at 
org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
at 
org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:74)
at 
org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
{noformat}

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Status: Patch Available  (was: Open)

Update the path to use test.tmp.dir and put the jar under /data/files/.

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Attachment: HIVE-13798.patch

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13798.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13798:

Fix Version/s: 2.1.0

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-13798) Fix the unit test failure org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu reassigned HIVE-13798:
---

Assignee: Aihua Xu

> Fix the unit test failure 
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
> 
>
> Key: HIVE-13798
> URL: https://issues.apache.org/jira/browse/HIVE-13798
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292347#comment-15292347
 ] 

ASF GitHub Bot commented on HIVE-9660:
--

GitHub user omalley opened a pull request:

https://github.com/apache/hive/pull/77

HIVE-9660 Add length to ORC indexes so that the reader knows how much to 
read.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/hive hive-9660

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/77.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #77


commit 014e9aaec1cb8f7257b997e953e6cc30d34a71cf
Author: Owen O'Malley 
Date:   2016-03-26T02:39:12Z

HIVE-11417. Move the ReaderImpl and RowReaderImpl to the ORC module,
by making shims for the row by row reader.

commit afda4610a8c1ed9fe3adc86c6fc1b08b5fdae7aa
Author: Owen O'Malley 
Date:   2016-05-13T21:44:34Z

HIVE-9660 Add length to ORC indexes so that the reader knows how much
to read.




> store end offset of compressed data for RG in RowIndex in ORC
> -
>
> Key: HIVE-9660
> URL: https://issues.apache.org/jira/browse/HIVE-9660
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-9660.01.patch, HIVE-9660.02.patch, 
> HIVE-9660.03.patch, HIVE-9660.04.patch, HIVE-9660.05.patch, 
> HIVE-9660.06.patch, HIVE-9660.07.patch, HIVE-9660.07.patch, 
> HIVE-9660.08.patch, HIVE-9660.09.patch, HIVE-9660.10.patch, 
> HIVE-9660.10.patch, HIVE-9660.11.patch, HIVE-9660.patch, HIVE-9660.patch, 
> owen-hive-9660.patch
>
>
> Right now the end offset is estimated, which in some cases results in tons of 
> extra data being read.
> We can add a separate array to RowIndex (positions_v2?) that stores number of 
> compressed buffers for each RG, or end offset, or something, to remove this 
> estimation magic



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HIVE-13354) Add ability to specify Compaction options per table and per request

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292324#comment-15292324
 ] 

Eugene Koifman edited comment on HIVE-13354 at 5/19/16 11:29 PM:
-

{quote} // intentionally set this high so that ttp1 will not trigger major 
compaction later on
   conf.setFloatVar(HiveConf.ConfVars.HIVE_COMPACTOR_DELTA_PCT_THRESHOLD, 
0.8f);
{quote}
could this be moved to where it's used - it's confusing at its current location

{quote}
   runWorker(conf);  // compact ttp2
runWorker(conf);  // compact ttp1
runCleaner(conf);
rsp = txnHandler.showCompact(new ShowCompactRequest());
Assert.assertEquals(2, rsp.getCompacts().size());
Assert.assertEquals("ttp2", 
rsp.getCompacts().get(0).getTablename());
Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(0).getState());
Assert.assertEquals("ttp1", 
rsp.getCompacts().get(1).getTablename());
Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(1).getState());
{quote}
The "ready for cleaning" seems suspicious after successful runCleaner()...  
Also, perhaps TxnStrore.CLEANING_RESPONSE would be better

{quote}
   // ttp1 has 0.8 for DELTA_PCT_THRESHOLD (from hive conf), whereas 
ttp2 has 0.5 (from tblproperties)
// so only ttp2 will trigger major compaction for the newly 
inserted row (actual pct: 0.66)
{quote}
this seems wrong.ttp2 had 5 rows which were Major compacted into a base.  
Now 2 more rows are added.  2/5 = 40%
Perhaps compaction is triggered because in this case ORC headers make up 99% of 
the file size.

bq. 949 Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(2).getState());
I would've expected this state to be TxnStore.SUCCEEDED_RESPONSE after 
runCleaner().  Why isn't it?

bq. 973 
Assert.assertTrue(job.get("hive.compactor.table.props").contains("orc.compress.size4:8192"));
Why "size4"?

{quote}
void compact(String dbname, String tableName, String partitionName, 
CompactionType type,
1440   Map tblproperties) throws TException;
1440
{quote}
This is public API change so should probably deprecate the method with old 
signature

{quote}
348 pStmt = dbConn.prepareStatement("insert into COMPLETED_COMPACTIONS(CC_ID, 
CC_DATABASE, CC_TABLE, CC_PARTITION, CC_STATE, CC_TYPE, CC_TBLPROPERTIES, 
CC_WORKER_ID, CC_START, CC_END, CC_RUN_AS, CC_HIGHEST_TXN_ID, CC_META_INFO, 
CC_HADOOP_JOB_ID) VALUES(?,?,?,?,?, ?,?,?,?,?, ?,?,?)");
{quote}
A new column is added here but the number of "?" is the same.  How does this 
work?

{quote}
714 rs = stmt.executeQuery("select cc_id, cc_database, cc_table, 
cc_partition, cc_state, " +
715 "cc_tblproperties from COMPLETED_COMPACTIONS order by 
cc_database, cc_table, " +
716 "cc_partition, cc_id desc");
{quote}
Why do you need to know cc_tblproperties in order to delete the entry from 
history?

etc

Clearly no tests were run on this patch.  In fact the SQL statement errors 
would cause all these methods to fail, which would explain why your new tests 
end up seeing unexpected status for various compaction operations.


was (Author: ekoifman):
{quote} // intentionally set this high so that ttp1 will not trigger major 
compaction later on
   conf.setFloatVar(HiveConf.ConfVars.HIVE_COMPACTOR_DELTA_PCT_THRESHOLD, 
0.8f);
{quote}
could this be moved to where it's used - it's confusing at its current location

{quote}
   runWorker(conf);  // compact ttp2
runWorker(conf);  // compact ttp1
runCleaner(conf);
rsp = txnHandler.showCompact(new ShowCompactRequest());
Assert.assertEquals(2, rsp.getCompacts().size());
Assert.assertEquals("ttp2", 
rsp.getCompacts().get(0).getTablename());
Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(0).getState());
Assert.assertEquals("ttp1", 
rsp.getCompacts().get(1).getTablename());
Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(1).getState());
{quote}
The "ready for cleaning" seems suspicious after successful runCleaner()...  
Also, perhaps TxnStrore.CLEANING_RESPONSE would be better

{quote}
   // ttp1 has 0.8 for DELTA_PCT_THRESHOLD (from hive conf), whereas 
ttp2 has 0.5 (from tblproperties)
// so only ttp2 will trigger major compaction for the newly 
inserted row (actual pct: 0.66)
{quote}
this seems wrong.ttp2 had 5 rows which were Major compacted into a base.  
Now 2 more rows are added.  2/5 = 40%
Perhaps compaction is triggered because in this case ORC headers make up 99% of 
the file size.

bq. 949 Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(2).getState());
I would've expected this state to be TxnStore.SUCCEEDED_RESPONSE after 

[jira] [Updated] (HIVE-13796) fix some tests on branch-1

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13796:

Status: Patch Available  (was: Open)

> fix some tests on branch-1
> --
>
> Key: HIVE-13796
> URL: https://issues.apache.org/jira/browse/HIVE-13796
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13796-branch-1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13731) LLAP: return LLAP token with the splits

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13731:

Status: Patch Available  (was: Reopened)

> LLAP: return LLAP token with the splits
> ---
>
> Key: HIVE-13731
> URL: https://issues.apache.org/jira/browse/HIVE-13731
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13731.patch, HIVE-13731.wo.13444-13675-13443.patch
>
>
> Need to return the token with the splits, then take it in LLAPIF and make 
> sure it's used when talking to LLAP



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13354) Add ability to specify Compaction options per table and per request

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292324#comment-15292324
 ] 

Eugene Koifman commented on HIVE-13354:
---

{quote} // intentionally set this high so that ttp1 will not trigger major 
compaction later on
   conf.setFloatVar(HiveConf.ConfVars.HIVE_COMPACTOR_DELTA_PCT_THRESHOLD, 
0.8f);
{quote}
could this be moved to where it's used - it's confusing at its current location

{quote}
   runWorker(conf);  // compact ttp2
runWorker(conf);  // compact ttp1
runCleaner(conf);
rsp = txnHandler.showCompact(new ShowCompactRequest());
Assert.assertEquals(2, rsp.getCompacts().size());
Assert.assertEquals("ttp2", 
rsp.getCompacts().get(0).getTablename());
Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(0).getState());
Assert.assertEquals("ttp1", 
rsp.getCompacts().get(1).getTablename());
Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(1).getState());
{quote}
The "ready for cleaning" seems suspicious after successful runCleaner()...  
Also, perhaps TxnStrore.CLEANING_RESPONSE would be better

{quote}
   // ttp1 has 0.8 for DELTA_PCT_THRESHOLD (from hive conf), whereas 
ttp2 has 0.5 (from tblproperties)
// so only ttp2 will trigger major compaction for the newly 
inserted row (actual pct: 0.66)
{quote}
this seems wrong.ttp2 had 5 rows which were Major compacted into a base.  
Now 2 more rows are added.  2/5 = 40%
Perhaps compaction is triggered because in this case ORC headers make up 99% of 
the file size.

bq. 949 Assert.assertEquals("ready for cleaning", 
rsp.getCompacts().get(2).getState());
I would've expected this state to be TxnStore.SUCCEEDED_RESPONSE after 
runCleaner().  Why isn't it?

bq. 973 
Assert.assertTrue(job.get("hive.compactor.table.props").contains("orc.compress.size4:8192"));
Why "size4"?

{quote}
void compact(String dbname, String tableName, String partitionName, 
CompactionType type,
1440   Map tblproperties) throws TException;
1440
{quote}
This is public API change so should probably deprecate the method with old 
signature

{quote}
348 pStmt = dbConn.prepareStatement("insert into COMPLETED_COMPACTIONS(CC_ID, 
CC_DATABASE, CC_TABLE, CC_PARTITION, CC_STATE, CC_TYPE, CC_TBLPROPERTIES, 
CC_WORKER_ID, CC_START, CC_END, CC_RUN_AS, CC_HIGHEST_TXN_ID, CC_META_INFO, 
CC_HADOOP_JOB_ID) VALUES(?,?,?,?,?, ?,?,?,?,?, ?,?,?)");
{quote}
A new column is added here but the number of "?" is the same.  How does this 
work?

{quote}
714 rs = stmt.executeQuery("select cc_id, cc_database, cc_table, 
cc_partition, cc_state, " +
715 "cc_tblproperties from COMPLETED_COMPACTIONS order by 
cc_database, cc_table, " +
716 "cc_partition, cc_id desc");
{quote}
Why do you need to know cc_tblproperties in order to delete the entry from 
history?

etc


> Add ability to specify Compaction options per table and per request
> ---
>
> Key: HIVE-13354
> URL: https://issues.apache.org/jira/browse/HIVE-13354
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>  Labels: TODOC2.1
> Attachments: HIVE-13354.1.patch, 
> HIVE-13354.1.withoutSchemaChange.patch
>
>
> Currently the are a few options that determine when automatic compaction is 
> triggered.  They are specified once for the warehouse.
> This doesn't make sense - some table may be more important and need to be 
> compacted more often.
> We should allow specifying these on per table basis.
> Also, compaction is an MR job launched from within the metastore.  There is 
> currently no way to control job parameters (like memory, for example) except 
> to specify it in hive-site.xml for metastore which means they are site wide.
> Should add a way to specify these per table (perhaps even per compaction if 
> launched via ALTER TABLE)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13783) No secondary prompt

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292304#comment-15292304
 ] 

Vihang Karajgaonkar commented on HIVE-13783:


Thanks for the review [~sircodesalot]

> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17 
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1> 
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and 
> pressing Enter, I get back the normal prompt. This easily makes me believe 
> I'm about to type a new command from scratch, e.g. a select query as in the 
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened 
> or anything like this, a differently looking secondary prompt should be 
> displayed rather than the normal prompt; as this is done in e.g. hive, 
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13796) fix some tests on branch-1

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13796:

Attachment: (was: HIVE-13796.patch)

> fix some tests on branch-1
> --
>
> Key: HIVE-13796
> URL: https://issues.apache.org/jira/browse/HIVE-13796
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13796-branch-1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13796) fix some tests on branch-1

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13796:

Attachment: HIVE-13796-branch-1.patch

> fix some tests on branch-1
> --
>
> Key: HIVE-13796
> URL: https://issues.apache.org/jira/browse/HIVE-13796
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13796-branch-1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13796) fix some tests on branch-1

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13796:

Attachment: HIVE-13796.patch

[~ashutoshc] can you take a look?

> fix some tests on branch-1
> --
>
> Key: HIVE-13796
> URL: https://issues.apache.org/jira/browse/HIVE-13796
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13796.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13725) ACID: Streaming API should synchronize calls when multiple threads use the same endpoint

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292258#comment-15292258
 ] 

Eugene Koifman commented on HIVE-13725:
---

Would it make sense to make a thread safe metastore client generally available? 
The "out of sequence response" issue keeps showing up periodically as new code 
is being added.  
Something like 
https://docs.oracle.com/javase/7/docs/api/java/util/Collections.html#synchronizedList(java.util.List)



> ACID: Streaming API should synchronize calls when multiple threads use the 
> same endpoint
> 
>
> Key: HIVE-13725
> URL: https://issues.apache.org/jira/browse/HIVE-13725
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Metastore, Transactions
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>  Labels: ACID, Streaming
>
> Currently, the streaming endpoint creates a metastore client which gets used 
> for RPC. The client itself is not internally thread safe. Therefore, the API 
> methods should provide the relevant synchronization so that the methods can 
> be called from different threads. A sample use case is as follows:
> 1. Thread 1 creates a streaming endpoint and opens a txn batch.
> 2. Thread 2 heartbeats the txn batch.
> With the current impl, this can result in an "out of sequence response", 
> since the response of the calls in thread1 might end up going to thread2 and 
> vice-versa.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13731) LLAP: return LLAP token with the splits

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13731:

Attachment: HIVE-13731.patch
HIVE-13731.wo.13444-13675-13443.patch

A small patch on top of the last 3 patches. [~sseth] [~jdere] fyi

> LLAP: return LLAP token with the splits
> ---
>
> Key: HIVE-13731
> URL: https://issues.apache.org/jira/browse/HIVE-13731
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13731.patch, HIVE-13731.wo.13444-13675-13443.patch
>
>
> Need to return the token with the splits, then take it in LLAPIF and make 
> sure it's used when talking to LLAP



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13448) LLAP: check ZK acls for ZKSM and fail if they are too permissive

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292179#comment-15292179
 ] 

Siddharth Seth commented on HIVE-13448:
---

+1

> LLAP: check ZK acls for ZKSM and fail if they are too permissive
> 
>
> Key: HIVE-13448
> URL: https://issues.apache.org/jira/browse/HIVE-13448
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13448.01.patch, HIVE-13448.02.patch, 
> HIVE-13448.patch
>
>
> Only the current user should have any access.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13708) Create table should verify datatypes supported by the serde

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292163#comment-15292163
 ] 

Hive QA commented on HIVE-13708:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804836/HIVE-13708.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 962 failed/errored test(s), 10030 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-union2.q-bucket4.q-and-12-more - did 
not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-cte_5.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_char2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_index
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_varchar2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_view_as_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_admin_almighty2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_cli_nonsql
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_show_grant
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19_inclause
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_change_schema
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_comments
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_compression_enabled
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_evolved_schemas
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_joins
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_nullable_fields
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_sanity_test
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_schema_evolution_native
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_schema_literal
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_case_sensitivity
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_input26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_gby_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join0

[jira] [Commented] (HIVE-13794) HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292155#comment-15292155
 ] 

Siddharth Seth commented on HIVE-13794:
---

+1

> HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits
> 
>
> Key: HIVE-13794
> URL: https://issues.apache.org/jira/browse/HIVE-13794
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13794.1.patch
>
>
> This option was being added in the test, but really should be set any time we 
> are generating the LLAP input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292078#comment-15292078
 ] 

Jesus Camacho Rodriguez commented on HIVE-13750:


Fails are unrelated, except 
{{org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_row_sequence}}
 for which I regenerated the .q file.

Pushed to master, thanks for reviewing [~ashutoshc]!

> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer 
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.1.0
>
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch, 
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted 
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO) 
> unconditionally adds an extra shuffle stage. If sort columns of previous 
> shuffle and partitioning columns of table match, reduce sink deduplication 
> optimizer removes extra shuffle stage, thus bringing down overhead to zero. 
> However, if they don’t match, we end up doing extra shuffle. This can be 
> improved since we can add table partition columns as a sort columns on 
> earlier shuffle and avoid this extra shuffle. This ensures that in cases 
> query already has a shuffle stage, we are not shuffling data again. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-13750:
---
   Resolution: Fixed
Fix Version/s: 2.1.0
   Status: Resolved  (was: Patch Available)

> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer 
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.1.0
>
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch, 
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted 
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO) 
> unconditionally adds an extra shuffle stage. If sort columns of previous 
> shuffle and partitioning columns of table match, reduce sink deduplication 
> optimizer removes extra shuffle stage, thus bringing down overhead to zero. 
> However, if they don’t match, we end up doing extra shuffle. This can be 
> improved since we can add table partition columns as a sort columns on 
> earlier shuffle and avoid this extra shuffle. This ensures that in cases 
> query already has a shuffle stage, we are not shuffling data again. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-13622) WriteSet tracking optimizations

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-13622.
---
   Resolution: Fixed
Fix Version/s: 2.1.0
   1.3.0

> WriteSet tracking optimizations
> ---
>
> Key: HIVE-13622
> URL: https://issues.apache.org/jira/browse/HIVE-13622
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, 
> HIVE-13622.4.patch, HIVE-13622.branch-1.patch
>
>
> HIVE-13395 solves the the lost update problem with some inefficiencies.
> 1. TxhHandler.OperationType is currently derived from LockType.  This doesn't 
>  distinguish between Update and Delete but would be useful.  See comments in 
> TxnHandler.  Should be able to pass in Insert/Update/Delete info from client 
> into TxnHandler.
> 2. TxnHandler.addDynamicPartitions() should know the OperationType as well 
> from the client.  It currently extrapolates it from TXN_COMPONENTS.  This 
> works but requires extra SQL statements and is thus less performant.  It will 
> not work multi-stmt txns.  See comments in the code.
> 3. TxnHandler.checkLock() see more comments around 
> "isPartOfDynamicPartitionInsert".  If TxnHandler knew whether it is being 
> called as part of an op running with dynamic partitions, it could be more 
> efficient.  In that case we don't have to write to TXN_COMPONENTS at all 
> during lock acquisition.  Conversely, if not running with DynPart then, we 
> can kill current txn on lock grant rather than wait until commit time.
> 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing 
> multiple rows into single SQL stmt (but with a limit for extreme cases)
> 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that 
> are only being read to TXN_COMPONENTS.   This is useless at best since read 
> op don't generate anything to compact.  For example, delete from T where t1 
> in (select c1 from C) - no reason to add C to txn_components but we do.
>  
> All of these require some Thrift changes
> Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11()
> Also see comments in 
> [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13622) WriteSet tracking optimizations

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13622:
--
Attachment: HIVE-13622.branch-1.patch

filed HIVE-13795 to followup on Item 3.

Committed to branch-1 and master
Thanks Alan for the review

> WriteSet tracking optimizations
> ---
>
> Key: HIVE-13622
> URL: https://issues.apache.org/jira/browse/HIVE-13622
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13622.2.patch, HIVE-13622.3.patch, 
> HIVE-13622.4.patch, HIVE-13622.branch-1.patch
>
>
> HIVE-13395 solves the the lost update problem with some inefficiencies.
> 1. TxhHandler.OperationType is currently derived from LockType.  This doesn't 
>  distinguish between Update and Delete but would be useful.  See comments in 
> TxnHandler.  Should be able to pass in Insert/Update/Delete info from client 
> into TxnHandler.
> 2. TxnHandler.addDynamicPartitions() should know the OperationType as well 
> from the client.  It currently extrapolates it from TXN_COMPONENTS.  This 
> works but requires extra SQL statements and is thus less performant.  It will 
> not work multi-stmt txns.  See comments in the code.
> 3. TxnHandler.checkLock() see more comments around 
> "isPartOfDynamicPartitionInsert".  If TxnHandler knew whether it is being 
> called as part of an op running with dynamic partitions, it could be more 
> efficient.  In that case we don't have to write to TXN_COMPONENTS at all 
> during lock acquisition.  Conversely, if not running with DynPart then, we 
> can kill current txn on lock grant rather than wait until commit time.
> 4. TxnHandler.addDynamicPartitions() - the insert stmt here should combing 
> multiple rows into single SQL stmt (but with a limit for extreme cases)
> 5. TxnHandler.enqueueLockWithRetry() - this currently adds components that 
> are only being read to TXN_COMPONENTS.   This is useless at best since read 
> op don't generate anything to compact.  For example, delete from T where t1 
> in (select c1 from C) - no reason to add C to txn_components but we do.
>  
> All of these require some Thrift changes
> Once done, re-enable TestDbTxnHandler2.testWriteSetTracking11()
> Also see comments in 
> [here|https://issues.apache.org/jira/browse/HIVE-13395?focusedCommentId=15271712=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15271712]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9798) Slash character in HDFS file is displayed as double Slash in Hive

[Gayathiri Duraikannu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292037#comment-15292037
 ] 

Gayathiri Duraikannu commented on HIVE-9798:


We are also having a similar issue in our project where the data is HDFS is 
having a single backslash but when queried with hive, double backslash is 
displayed. As this field is a varchar(60) and having 60 length, the last 
character is getting truncated. Hive is considering the escape backslash 
internally added by the hive program as one character and ignoring the last 
character in the actual data.

HDFS: 
Org_!@#\ab%^()_test1tt2t928dbb73-e4e4-4c7c-a948-0391ba34f8cf

Hive: 
Org_!@#\\ab%^()_test1tt2t928dbb73-e4e4-4c7c-a948-0391ba34f8c

Clients are missing the 'f' at the end when viewing the data with hive. 

when the data is selected and written to a file, it's displaying correctly. As 
a work around we are casting the column to String. 

Please let me know if there is any permanent solution for it. 


> Slash character in HDFS file is displayed as double Slash in Hive
> -
>
> Key: HIVE-9798
> URL: https://issues.apache.org/jira/browse/HIVE-9798
> Project: Hive
>  Issue Type: Bug
>Reporter: Anitha
>
> Single slash character in source is displayed as double slash in HIVE.
> Example:
> Oracle Source: BOB RICHARD TOY\N AUGUSTA
> Hive = BOB RICHARD TOY\\N AUGUSTA
> Note: HDFS file has correct data, single slash only. Also, Impala displays 
> the data correctly, issue is with Hive display only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HIVE-13741) TxnHandler.enqueueLockWithRetry() - optimize sql

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13741:
--
Comment: was deleted

(was: also 
TxnHandler.enqueueLockWithRetry() - this currently adds components that are 
only being read to TXN_COMPONENTS. This is useless at best since read op don't 
generate anything to compact. For example, delete from T where t1 in (select c1 
from C) - no reason to add C to txn_components but we do.
)

> TxnHandler.enqueueLockWithRetry() - optimize  sql
> -
>
> Key: HIVE-13741
> URL: https://issues.apache.org/jira/browse/HIVE-13741
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
> (possibly) multiple statements for each.  Could easily generate 1 statement 
> for each table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HIVE-13741) TxnHandler.enqueueLockWithRetry() - optimize sql

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13741:
--
Comment: was deleted

(was: replace previous comment with this one

TxnHandler.addDynamicPartitions() - the insert stmt here should combing 
multiple rows into single SQL stmt (but with a limit for extreme cases))

> TxnHandler.enqueueLockWithRetry() - optimize  sql
> -
>
> Key: HIVE-13741
> URL: https://issues.apache.org/jira/browse/HIVE-13741
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
> (possibly) multiple statements for each.  Could easily generate 1 statement 
> for each table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13741) TxnHandler.enqueueLockWithRetry() - optimize sql

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13741:
--
Description: 
TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
(possibly) multiple statements for each.  Could easily generate 1 statement for 
each table.

TxnHandler.addDynamicPartitions() - the insert stmt here should combing 
multiple rows into single SQL stmt (but with a limit for extreme cases)


  was:TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
(possibly) multiple statements for each.  Could easily generate 1 statement for 
each table.


> TxnHandler.enqueueLockWithRetry() - optimize  sql
> -
>
> Key: HIVE-13741
> URL: https://issues.apache.org/jira/browse/HIVE-13741
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
> (possibly) multiple statements for each.  Could easily generate 1 statement 
> for each table.
> TxnHandler.addDynamicPartitions() - the insert stmt here should combing 
> multiple rows into single SQL stmt (but with a limit for extreme cases)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13741) TxnHandler.enqueueLockWithRetry() - optimize sql

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13741:
--
Assignee: (was: Eugene Koifman)

> TxnHandler.enqueueLockWithRetry() - optimize  sql
> -
>
> Key: HIVE-13741
> URL: https://issues.apache.org/jira/browse/HIVE-13741
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
> (possibly) multiple statements for each.  Could easily generate 1 statement 
> for each table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13741) TxnHandler.enqueueLockWithRetry() - optimize sql

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-13741:
--
Summary: TxnHandler.enqueueLockWithRetry() - optimize  sql  (was: 
TxnHandler.enqueueLockWithRetry() - optimizer  sql)

> TxnHandler.enqueueLockWithRetry() - optimize  sql
> -
>
> Key: HIVE-13741
> URL: https://issues.apache.org/jira/browse/HIVE-13741
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>   Original Estimate: 3h
>  Remaining Estimate: 3h
>
> TxnHandler.enqueueLockWithRetry()  does SQL insert into 2 tables using 
> (possibly) multiple statements for each.  Could easily generate 1 statement 
> for each table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-13779) Fraction of second is lost when you change column from timestamp to string

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere resolved HIVE-13779.
---
Resolution: Not A Bug

I think the behavior has always been like this - it looks like 
TimestampWritable.toString() (which is used when converting to String) has 
logic to remove the trailing 0 from a timestamp value if the fractional portion 
is 0, whereas the java.sql.Timestamp type will keep a trailing 0 even if there 
is no fractional portion (1.0, 99.0, etc). We aren't losing any data here.

> Fraction of second is lost when you change column from timestamp to string
> --
>
> Key: HIVE-13779
> URL: https://issues.apache.org/jira/browse/HIVE-13779
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Takahiko Saito
>
> I have this table:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select s, ts from schemaevo 
> limit 10;
> +---++--+
> | s |   ts   |
> +---++--+
> | david falkner | 2076-08-03 22:22:45.0  |
> | oscar miller  | 2027-05-23 18:51:23.0  |
> | david xylophone   | 2031-02-07 13:02:38.0  |
> | katie allen   | 2035-04-22 08:02:07.0  |
> | zach johnson  | 2074-02-21 18:25:58.0  |
> | mike brown| 2029-05-23 04:09:04.0  |
> | luke garcia   | 2059-03-14 12:37:41.0  |
> | xavier garcia | 2025-04-22 11:23:44.0  |
> | holly laertes | 2037-01-20 08:02:40.0  |
> | xavier underhill  | 2072-08-16 07:41:05.0  |
> +---++--+
> {noformat}
> Then run the followings:
> {noformat}
> set hive.metastore.disallow.incompatible.col.type.changes = false;
> drop table if exists 
> shcemaevo_vectorization_true_fileformat_orc_timestamp_string;
> create table shcemaevo_vectorization_true_fileformat_orc_timestamp_string 
> stored as orc as select s, ts from schemaevo;
> alter table shcemaevo_vectorization_true_fileformat_orc_timestamp_string 
> change column ts ts string;
> {noformat}
> The below shows that the changed column lose fraction of second:
> {noformat}
> 0: jdbc:hive2://os-r6-0517-hiveserver2-1-1.op> select * from 
> shcemaevo_vectorization_true_fileformat_orc_timestamp_string limit 10;
> +-+--+--+
> | shcemaevo_vectorization_true_fileformat_orc_timestamp_string.s  | 
> shcemaevo_vectorization_true_fileformat_orc_timestamp_string.ts  |
> +-+--+--+
> | david falkner   | 
> 2076-08-03 22:22:45  |
> | oscar miller| 
> 2027-05-23 18:51:23  |
> | david xylophone | 
> 2031-02-07 13:02:38  |
> | katie allen | 
> 2035-04-22 08:02:07  |
> | zach johnson| 
> 2074-02-21 18:25:58  |
> | mike brown  | 
> 2029-05-23 04:09:04  |
> | luke garcia | 
> 2059-03-14 12:37:41  |
> | xavier garcia   | 
> 2025-04-22 11:23:44  |
> | holly laertes   | 
> 2037-01-20 08:02:40  |
> | xavier underhill| 
> 2072-08-16 07:41:05  |
> +-+--+--+
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292007#comment-15292007
 ] 

Wei Zheng commented on HIVE-13249:
--

Test failures are unrelated.

> Hard upper bound on number of open transactions
> ---
>
> Key: HIVE-13249
> URL: https://issues.apache.org/jira/browse/HIVE-13249
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13249.1.patch, HIVE-13249.10.patch, 
> HIVE-13249.11.patch, HIVE-13249.2.patch, HIVE-13249.3.patch, 
> HIVE-13249.4.patch, HIVE-13249.5.patch, HIVE-13249.6.patch, 
> HIVE-13249.7.patch, HIVE-13249.8.patch, HIVE-13249.9.patch
>
>
> We need to have a safeguard by adding an upper bound for open transactions to 
> avoid huge number of open-transaction requests, usually due to improper 
> configuration of clients such as Storm.
> Once that limit is reached, clients will start failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13562:

Fix Version/s: 1.3.0

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291929#comment-15291929
 ] 

Matt McCline commented on HIVE-13562:
-

Committed to branch-1.

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291912#comment-15291912
 ] 

Hive QA commented on HIVE-13750:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804916/HIVE-13750.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 88 failed/errored test(s), 10026 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join1.q-schema_evol_text_vec_mapwork_part_all_complex.q-vector_complex_join.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_row_sequence
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_merge1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_smb_cache
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join0
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestFirstInFirstOutComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks

[jira] [Updated] (HIVE-13792) Show create table should not show stats info in the table properties

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13792:

Status: Patch Available  (was: Open)

Attached the patch-1: the patch will filter out the stats parameters from 'show 
create table' command output since those should not be part of the create table 
parameters.

> Show create table should not show stats info in the table properties
> 
>
> Key: HIVE-13792
> URL: https://issues.apache.org/jira/browse/HIVE-13792
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13792.1.patch
>
>
> From the test 
> org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 
> failure, we are printing table stats in show create table parameters. This 
> info should be skipped since it would be incorrect when you just copy them to 
> create a table. 
> {noformat}
> CREATE EXTERNAL TABLE `hbase_table_1_like`(
>   `key` int COMMENT 'It is a column key',
>   `value` string COMMENT 'It is the column string value')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
>   'hbase.columns.mapping'='cf:string',
>   'serialization.format'='1')
> TBLPROPERTIES (
>   'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
>   'hbase.table.name'='hbase_table_0',
>   'numFiles'='0',
>   'numRows'='0',
>   'rawDataSize'='0',
>   'totalSize'='0',
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13792) Show create table should not show stats info in the table properties

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13792:

Attachment: HIVE-13792.1.patch

> Show create table should not show stats info in the table properties
> 
>
> Key: HIVE-13792
> URL: https://issues.apache.org/jira/browse/HIVE-13792
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13792.1.patch
>
>
> From the test 
> org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 
> failure, we are printing table stats in show create table parameters. This 
> info should be skipped since it would be incorrect when you just copy them to 
> create a table. 
> {noformat}
> CREATE EXTERNAL TABLE `hbase_table_1_like`(
>   `key` int COMMENT 'It is a column key',
>   `value` string COMMENT 'It is the column string value')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
>   'hbase.columns.mapping'='cf:string',
>   'serialization.format'='1')
> TBLPROPERTIES (
>   'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
>   'hbase.table.name'='hbase_table_0',
>   'numFiles'='0',
>   'numRows'='0',
>   'rawDataSize'='0',
>   'totalSize'='0',
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13792) Show create table should not show stats info in the table properties

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13792:

Description: 
>From the test 
>org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 
>failure, we are printing table stats in show create table parameters. This 
>info should be skipped since it would be incorrect when you just copy them to 
>create a table. 

{noformat}
CREATE EXTERNAL TABLE `hbase_table_1_like`(
  `key` int COMMENT 'It is a column key',
  `value` string COMMENT 'It is the column string value')
ROW FORMAT SERDE
  'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY
  'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
  'hbase.columns.mapping'='cf:string',
  'serialization.format'='1')
TBLPROPERTIES (
  'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
  'hbase.table.name'='hbase_table_0',
  'numFiles'='0',
  'numRows'='0',
  'rawDataSize'='0',
  'totalSize'='0',
{noformat}

  was:
>From the test 
>org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 
>failure, we are printing table stats in show create table parameters. This 
>info should be skipped since it would be incorrect when you just copy them to 
>create a table. And also the format for TBLPROPERTIES is not well formed.

{noformat}
CREATE EXTERNAL TABLE `hbase_table_1_like`(
  `key` int COMMENT 'It is a column key',
  `value` string COMMENT 'It is the column string value')
ROW FORMAT SERDE
  'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY
  'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
  'hbase.columns.mapping'='cf:string',
  'serialization.format'='1')
TBLPROPERTIES (
  'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
  'hbase.table.name'='hbase_table_0',
  'numFiles'='0',
  'numRows'='0',
  'rawDataSize'='0',
  'totalSize'='0',
{noformat}


> Show create table should not show stats info in the table properties
> 
>
> Key: HIVE-13792
> URL: https://issues.apache.org/jira/browse/HIVE-13792
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>
> From the test 
> org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries 
> failure, we are printing table stats in show create table parameters. This 
> info should be skipped since it would be incorrect when you just copy them to 
> create a table. 
> {noformat}
> CREATE EXTERNAL TABLE `hbase_table_1_like`(
>   `key` int COMMENT 'It is a column key',
>   `value` string COMMENT 'It is the column string value')
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.hbase.HBaseSerDe'
> STORED BY
>   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
>   'hbase.columns.mapping'='cf:string',
>   'serialization.format'='1')
> TBLPROPERTIES (
>   'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
>   'hbase.table.name'='hbase_table_0',
>   'numFiles'='0',
>   'numRows'='0',
>   'rawDataSize'='0',
>   'totalSize'='0',
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13794) HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-13794:
--
Attachment: HIVE-13794.1.patch

> HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits
> 
>
> Key: HIVE-13794
> URL: https://issues.apache.org/jira/browse/HIVE-13794
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13794.1.patch
>
>
> This option was being added in the test, but really should be set any time we 
> are generating the LLAP input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13794) HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-13794:
--
Status: Patch Available  (was: Open)

> HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits
> 
>
> Key: HIVE-13794
> URL: https://issues.apache.org/jira/browse/HIVE-13794
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13794.1.patch
>
>
> This option was being added in the test, but really should be set any time we 
> are generating the LLAP input splits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13787) LLAP: bug in recent security patches

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-13787:

Status: Patch Available  (was: Open)

> LLAP: bug in recent security patches
> 
>
> Key: HIVE-13787
> URL: https://issues.apache.org/jira/browse/HIVE-13787
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13787.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13759) LlapTaskUmbilicalExternalClient should be closed by the record reader

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-13759:
--
Attachment: HIVE-13759.1.patch

> LlapTaskUmbilicalExternalClient should be closed by the record reader
> -
>
> Key: HIVE-13759
> URL: https://issues.apache.org/jira/browse/HIVE-13759
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-13759.1.patch
>
>
> The umbilical external client (and the server socket it creates) doesn't look 
> like it's getting closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13759) LlapTaskUmbilicalExternalClient should be closed by the record reader

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-13759:
--
Attachment: (was: HIVE-13759.1.patch)

> LlapTaskUmbilicalExternalClient should be closed by the record reader
> -
>
> Key: HIVE-13759
> URL: https://issues.apache.org/jira/browse/HIVE-13759
> Project: Hive
>  Issue Type: Sub-task
>  Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
>
> The umbilical external client (and the server socket it creates) doesn't look 
> like it's getting closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13782) Compile async query asynchronously

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291518#comment-15291518
 ] 

Xuefu Zhang commented on HIVE-13782:


+1

> Compile async query asynchronously
> --
>
> Key: HIVE-13782
> URL: https://issues.apache.org/jira/browse/HIVE-13782
> Project: Hive
>  Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-13782.1.patch
>
>
> Currently, when an async query is submitted to HS2, HS2 does the preparation 
> synchronously. One of the preparation step is to compile the query, which may 
> take some time. It will be helpful to provide an option to do the compilation 
> asynchronously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12866) Allow ReduceSinkDeDuplication to kick in when there are constant keys

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291492#comment-15291492
 ] 

Ashutosh Chauhan commented on HIVE-12866:
-

As a pre-requisite of this _bucket_number_ column introduced by SPDO should be 
created as of type ExprNodeColumnDesc. Currently, its incorrectly declared as 
ExprNodeConstantDesc.

> Allow ReduceSinkDeDuplication to kick in when there are constant keys 
> --
>
> Key: HIVE-12866
> URL: https://issues.apache.org/jira/browse/HIVE-12866
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>
> Currently in ReduceSinkDeDuplication.checkExprs() we have the following check 
> which prevents the optimization from kicking in as soon as we hit constant 
> key. We should lift this restriction as much as possible.
> {code}
> private Integer checkExprs(List ckeys, List 
> pkeys,
> ReduceSinkOperator cRS, ReduceSinkOperator pRS) throws 
> SemanticException {
>   // If ckeys or pkeys have constant node expressions avoid the merge.
>   for (ExprNodeDesc ck : ckeys) {
> if (ck instanceof ExprNodeConstantDesc) {
>   return null;
> }
>   }
>   for (ExprNodeDesc pk : pkeys) {
> if (pk instanceof ExprNodeConstantDesc) {
>   return null;
> }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13750) Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer when possible

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291490#comment-15291490
 ] 

Ashutosh Chauhan commented on HIVE-13750:
-

+1

> Avoid additional shuffle stage created by Sorted Dynamic Partition Optimizer 
> when possible
> --
>
> Key: HIVE-13750
> URL: https://issues.apache.org/jira/browse/HIVE-13750
> Project: Hive
>  Issue Type: Improvement
>  Components: Physical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13750.01.patch, HIVE-13750.02.patch, 
> HIVE-13750.patch, HIVE-13750.patch
>
>
> Extend ReduceDedup to remove additional shuffle stage created by sorted 
> dynamic partition optimizer when possible, thus avoiding unnecessary work.
> By [~ashutoshc]:
> {quote}
> Currently, if config is on Sorted Dynamic Partition Optimizer (SDPO) 
> unconditionally adds an extra shuffle stage. If sort columns of previous 
> shuffle and partitioning columns of table match, reduce sink deduplication 
> optimizer removes extra shuffle stage, thus bringing down overhead to zero. 
> However, if they don’t match, we end up doing extra shuffle. This can be 
> improved since we can add table partition columns as a sort columns on 
> earlier shuffle and avoid this extra shuffle. This ensures that in cases 
> query already has a shuffle stage, we are not shuffling data again. 
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-10176:

Target Version/s: 2.1.0

> skip.header.line.count causes values to be skipped when performing insert 
> values
> 
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch, 
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch, 
> HIVE-10176.14.patch, HIVE-10176.15.patch, HIVE-10176.16.patch, 
> HIVE-10176.2.patch, HIVE-10176.3.patch, HIVE-10176.4.patch, 
> HIVE-10176.5.patch, HIVE-10176.6.patch, HIVE-10176.7.patch, 
> HIVE-10176.8.patch, HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES 
> ("skip.header.line.count"="1") the first value listed is also skipped. 
> create table test (row int, name string) TBLPROPERTIES 
> ("skip.header.line.count"="1"); 
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10176) skip.header.line.count causes values to be skipped when performing insert values

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-10176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291476#comment-15291476
 ] 

Ashutosh Chauhan commented on HIVE-10176:
-

[~lmephistol] Are any of the reported failures related to patch?

> skip.header.line.count causes values to be skipped when performing insert 
> values
> 
>
> Key: HIVE-10176
> URL: https://issues.apache.org/jira/browse/HIVE-10176
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.0.0, 1.2.1
>Reporter: Wenbo Wang
>Assignee: Vladyslav Pavlenko
> Fix For: 2.0.0
>
> Attachments: HIVE-10176.1.patch, HIVE-10176.10.patch, 
> HIVE-10176.11.patch, HIVE-10176.12.patch, HIVE-10176.13.patch, 
> HIVE-10176.14.patch, HIVE-10176.15.patch, HIVE-10176.16.patch, 
> HIVE-10176.2.patch, HIVE-10176.3.patch, HIVE-10176.4.patch, 
> HIVE-10176.5.patch, HIVE-10176.6.patch, HIVE-10176.7.patch, 
> HIVE-10176.8.patch, HIVE-10176.9.patch, data
>
>
> When inserting values in to tables with TBLPROPERTIES 
> ("skip.header.line.count"="1") the first value listed is also skipped. 
> create table test (row int, name string) TBLPROPERTIES 
> ("skip.header.line.count"="1"); 
> load data local inpath '/root/data' into table test;
> insert into table test values (1, 'a'), (2, 'b'), (3, 'c');
> (1, 'a') isn't inserted into the table. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291475#comment-15291475
 ] 

Matt McCline commented on HIVE-13562:
-

Committed to master.

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Fix For: 2.1.0
>
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13562:

Fix Version/s: 2.1.0

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Fix For: 2.1.0
>
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13562:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Fix For: 2.1.0
>
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291473#comment-15291473
 ] 

Matt McCline commented on HIVE-13562:
-

HIVE-13562.04.patch passed on an Internal PTest framework (the test failures 
were unrelated).

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13562) Enable vector bridge for all non-vectorized udfs

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13562:

Attachment: HIVE-13562.04.patch

> Enable vector bridge for all non-vectorized udfs
> 
>
> Key: HIVE-13562
> URL: https://issues.apache.org/jira/browse/HIVE-13562
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Ashutosh Chauhan
>Assignee: Matt McCline
> Attachments: HIVE-13562.01.patch, HIVE-13562.02.patch, 
> HIVE-13562.03.patch, HIVE-13562.04.patch
>
>
> Mechanism already exists for this via {{VectorUDFAdaptor}} but we have 
> arbitrarily hand picked few udfs to go through it. I think we should enable 
> this by default for all udfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-13502:
-
Attachment: HIVE-13502.5.patch

Re-attaching the same patch as the prior did not get picked up by the 
pre-commit build. 

> Beeline doesnt support session parameters in JDBC URL as documentation states.
> --
>
> Key: HIVE-13502
> URL: https://issues.apache.org/jira/browse/HIVE-13502
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, 
> HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, 
> HIVE-13502.5.patch, HIVE-13502.patch
>
>
> https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs
> documents that sessions variables like credentials etc are accepted as part 
> of the URL. However, Beeline does not support such URLs today.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-13502:
-
Status: Patch Available  (was: Open)

> Beeline doesnt support session parameters in JDBC URL as documentation states.
> --
>
> Key: HIVE-13502
> URL: https://issues.apache.org/jira/browse/HIVE-13502
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, 
> HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, 
> HIVE-13502.5.patch, HIVE-13502.patch
>
>
> https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs
> documents that sessions variables like credentials etc are accepted as part 
> of the URL. However, Beeline does not support such URLs today.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-13502:
-
Attachment: (was: HIVE-13502.5.patch)

> Beeline doesnt support session parameters in JDBC URL as documentation states.
> --
>
> Key: HIVE-13502
> URL: https://issues.apache.org/jira/browse/HIVE-13502
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, 
> HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, HIVE-13502.patch
>
>
> https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs
> documents that sessions variables like credentials etc are accepted as part 
> of the URL. However, Beeline does not support such URLs today.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13502) Beeline doesnt support session parameters in JDBC URL as documentation states.

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-13502:
-
Status: Open  (was: Patch Available)

> Beeline doesnt support session parameters in JDBC URL as documentation states.
> --
>
> Key: HIVE-13502
> URL: https://issues.apache.org/jira/browse/HIVE-13502
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13502.1.patch, HIVE-13502.2.patch, 
> HIVE-13502.3.patch, HIVE-13502.3.patch, HIVE-13502.4.patch, HIVE-13502.patch
>
>
> https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLs
> documents that sessions variables like credentials etc are accepted as part 
> of the URL. However, Beeline does not support such URLs today.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-13791) Fix failure Unit Test TestHiveSessionImpl.testLeakOperationHandle

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-13791.
-
   Resolution: Duplicate
Fix Version/s: 2.1.0

> Fix  failure Unit Test TestHiveSessionImpl.testLeakOperationHandle
> --
>
> Key: HIVE-13791
> URL: https://issues.apache.org/jira/browse/HIVE-13791
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HIVE-13791.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291356#comment-15291356
 ] 

Hive QA commented on HIVE-13249:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804730/HIVE-13249.11.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 79 failed/errored test(s), 10034 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-union2.q-bucket4.q-and-12-more - did 
not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_pipe
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_self_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_data_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_date_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_aggregate
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_casts
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks

[jira] [Commented] (HIVE-13520) Don't allow any test to run for longer than 60minutes in the ptest setup

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291354#comment-15291354
 ] 

Ashutosh Chauhan commented on HIVE-13520:
-

+1

> Don't allow any test to run for longer than 60minutes in the ptest setup
> 
>
> Key: HIVE-13520
> URL: https://issues.apache.org/jira/browse/HIVE-13520
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-13520.01.txt, HIVE-13520.02.txt, HIVE-13520.03.patch
>
>
> Current timeout for batches is 2hours. This needs to be lowered. 1hour may be 
> too much as well. We can start with this, and reduce timeouts further.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13786:

   Resolution: Fixed
Fix Version/s: 2.1.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks Ashutosh for reviewing.

> Fix the unit test failure 
> org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
> -
>
> Key: HIVE-13786
> URL: https://issues.apache.org/jira/browse/HIVE-13786
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13786.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13791) Fix failure Unit Test TestHiveSessionImpl.testLeakOperationHandle

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291344#comment-15291344
 ] 

Ashutosh Chauhan commented on HIVE-13791:
-

Dupe of HIVE-13786

> Fix  failure Unit Test TestHiveSessionImpl.testLeakOperationHandle
> --
>
> Key: HIVE-13791
> URL: https://issues.apache.org/jira/browse/HIVE-13791
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-13791.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13789) Repeatedly checking configuration in TextRecordWriter/Reader hurts performance

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291332#comment-15291332
 ] 

Xuefu Zhang commented on HIVE-13789:


+1

> Repeatedly checking configuration in TextRecordWriter/Reader hurts performance
> --
>
> Key: HIVE-13789
> URL: https://issues.apache.org/jira/browse/HIVE-13789
> Project: Hive
>  Issue Type: Improvement
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HIVE-13789.1.patch
>
>
> We check configuration to decide whether to escape certain characters each 
> time write/read a record for custom scripts.
> In our benchmark this becomes a hot spot method. And fixing it improves the 
> execution of the custom script by 7% (3TB TPCx-BB dataset).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291285#comment-15291285
 ] 

Ashutosh Chauhan commented on HIVE-13786:
-

+1 yeah.. lets push it without waiting for a run.

> Fix the unit test failure 
> org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
> -
>
> Key: HIVE-13786
> URL: https://issues.apache.org/jira/browse/HIVE-13786
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13786.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13783) No secondary prompt

[Reuben Kuhnert (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291275#comment-15291275
 ] 

Reuben Kuhnert commented on HIVE-13783:
---

LGTM:
/
{code}
beeline> this
this
. . . .> seems
seems
. . . .> to
to
. . . .> work
work
. . . .> ;
;
beeline> No current connection
{code}

> No secondary prompt
> ---
>
> Key: HIVE-13783
> URL: https://issues.apache.org/jira/browse/HIVE-13783
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-13783.01.patch
>
>
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
> 0: jdbc:hive2://localhost:1> select * from foo;
> Error: Error while compiling statement: FAILED: ParseException line 2:17 
> character '' not supported here (state=42000,code=4)
> 0: jdbc:hive2://localhost:1> 
> {noformat}
> After (accidentally) entering a lonely quote character on its own line and 
> pressing Enter, I get back the normal prompt. This easily makes me believe 
> I'm about to type a new command from scratch, e.g. a select query as in the 
> example, which ends up not working due to parsing error.
> Expected behavior: When a previous command is continued, or a quote is opened 
> or anything like this, a differently looking secondary prompt should be 
> displayed rather than the normal prompt; as this is done in e.g. hive, 
> impala, mysql, bash..., e.g.:
> {noformat}
> # beeline -u jdbc:hive2://localhost:1
> [...]
> Beeline version 1.1.0-cdh5.4.5 by Apache Hive
> 0: jdbc:hive2://localhost:1> "
>> ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split

[Svetozar Ivanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Svetozar Ivanov updated HIVE-13584:
---
Attachment: HIVE-13584.patch

> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
>  Labels: HBase, PreSplit
> Fix For: 2.0.1
>
> Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler 
> with specified pre-split strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split

[Svetozar Ivanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Svetozar Ivanov updated HIVE-13584:
---
Status: Patch Available  (was: Open)

> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
>  Labels: HBase, PreSplit
> Fix For: 2.0.1
>
> Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler 
> with specified pre-split strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split

[Svetozar Ivanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Svetozar Ivanov updated HIVE-13584:
---
Status: Open  (was: Patch Available)

> HBaseStorageHandler should support table pre-split
> --
>
> Key: HIVE-13584
> URL: https://issues.apache.org/jira/browse/HIVE-13584
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
>  Labels: HBase, PreSplit
> Fix For: 2.0.1
>
> Attachments: HIVE-13584-branch-2.0.patch
>
>
> Currently is not possible creation of HBase table via HBaseStorageHandler 
> with specified pre-split strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler

[Svetozar Ivanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Svetozar Ivanov updated HIVE-11233:
---
Attachment: HIVE-11233.patch

> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
>  Issue Type: New Feature
>  Components: HBase Handler
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
>  Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch, 
> HIVE-11233-branch-2.0.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary 
> sortable key and values. It is necessary when given HBase table is used for 
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays 
> read or written by Hive will be compatible with binary sortable format used 
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly 
> officially provided documentation - 
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor 
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, 
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, 
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, 
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, 
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, 
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, 
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + 
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler

[Svetozar Ivanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Svetozar Ivanov updated HIVE-11233:
---
Status: Patch Available  (was: Open)

> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
>  Issue Type: New Feature
>  Components: HBase Handler
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
>  Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch, 
> HIVE-11233-branch-2.0.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary 
> sortable key and values. It is necessary when given HBase table is used for 
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays 
> read or written by Hive will be compatible with binary sortable format used 
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly 
> officially provided documentation - 
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor 
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, 
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, 
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, 
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, 
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, 
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, 
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + 
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler

[Svetozar Ivanov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Svetozar Ivanov updated HIVE-11233:
---
Status: Open  (was: Patch Available)

> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
>  Issue Type: New Feature
>  Components: HBase Handler
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
>  Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch, HIVE-11233-branch-2.0.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary 
> sortable key and values. It is necessary when given HBase table is used for 
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays 
> read or written by Hive will be compatible with binary sortable format used 
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly 
> officially provided documentation - 
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor 
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME, 
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, 
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING, 
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS, 
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS, 
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY, 
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " + 
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291142#comment-15291142
 ] 

Aihua Xu commented on HIVE-13786:
-

[~ashutoshc] and [~ctang.ma] Try to fix the recent test failures since it's 
hard to tell which ones are related.

Can you take a look at the simple test fix? Can we push it without going 
through the build process? I have tested locally.

> Fix the unit test failure 
> org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
> -
>
> Key: HIVE-13786
> URL: https://issues.apache.org/jira/browse/HIVE-13786
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13786.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-13786:

Comment: was deleted

(was: [~ashutoshc] and [~ctang.ma] Try to fix the recent test failures since 
it's hard to tell which ones are related.

Can you take a look at the simple test fix? Can we push it without going 
through the build process? I have tested locally.)

> Fix the unit test failure 
> org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
> -
>
> Key: HIVE-13786
> URL: https://issues.apache.org/jira/browse/HIVE-13786
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-13786.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291077#comment-15291077
 ] 

Hive QA commented on HIVE-13068:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12804789/HIVE-13068.05.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 148 failed/errored test(s), 10026 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-tez_dynpart_hashjoin_3.q-orc_vectorization_ppd.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_union_group_by.q-vector_auto_smb_mapjoin_14.q-union_fast_stats.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more - 
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_const
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_when_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_cond_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_eq_with_case_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_when
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_join_preds
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_nonacid_from_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lineage3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_query_oneskew_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_where
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_case
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_udf_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_allchildsarenull
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_basic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quotedid_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25

[jira] [Updated] (HIVE-13791) Fix failure Unit Test TestHiveSessionImpl.testLeakOperationHandle

[Nemon Lou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nemon Lou updated HIVE-13791:
-
Attachment: HIVE-13791.patch

I have run TestHiveSessionImpl successfully from local . 

> Fix  failure Unit Test TestHiveSessionImpl.testLeakOperationHandle
> --
>
> Key: HIVE-13791
> URL: https://issues.apache.org/jira/browse/HIVE-13791
> Project: Hive
>  Issue Type: Test
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Nemon Lou
>Assignee: Nemon Lou
>Priority: Minor
> Attachments: HIVE-13791.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13757) Add PerfLog information to genLogicalPlan in CalcitePlanner

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290958#comment-15290958
 ] 

Jesus Camacho Rodriguez commented on HIVE-13757:


[~ashutoshc], log looks like this:

{noformat}
...
2016-05-19T04:25:14,196 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogBegin(130)) - 
2016-05-19T04:25:14,196 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(878)) - DDL: struct src { 
string key, string value}
2016-05-19T04:25:14,197 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
parse.CalcitePlanner (CalcitePlanner.java:genLogicalPlan(3299)) - Created Plan 
for Query Block t1_v
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogEnd(170)) - 
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogBegin(130)) - 
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogEnd(170)) - 
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogBegin(130)) - 
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogEnd(170)) - 
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: 
log.PerfLogger (PerfLogger.java:PerfLogBegin(130)) - 
...
{noformat}

If we want to obtain more meaningful information e.g. recursive methods, 
multiple calls to same method, etc., PerfLogger infra should be extended. For 
instance, allow to aggregate the time spent in a certain method using keys to 
recognize them. Then at the end of query execution, dump all that measurements. 
Or even at the end of a given session.

> Add PerfLog information to genLogicalPlan in CalcitePlanner
> ---
>
> Key: HIVE-13757
> URL: https://issues.apache.org/jira/browse/HIVE-13757
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-13757.patch
>
>
> This will help us to have more accurate measurements of the different methods 
> used to generate the RelNode tree for the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-9795) Add rule to insert not null filtering on join keys

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez resolved HIVE-9795.
---
Resolution: Duplicate

Already pushed this rule as part of the work in HIVE-9132. Closing as duplicate.

> Add rule to insert not null filtering on join keys
> --
>
> Key: HIVE-9795
> URL: https://issues.apache.org/jira/browse/HIVE-9795
> Project: Hive
>  Issue Type: Bug
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-9795.patch
>
>
> Adding a Calcite rule that introduces _not null_ filtering on join keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11049) With Clause should cache data & reuse

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-11049:
---
Fix Version/s: (was: 1.3.0)

> With Clause should cache data & reuse 
> --
>
> Key: HIVE-11049
> URL: https://issues.apache.org/jira/browse/HIVE-11049
> Project: Hive
>  Issue Type: Improvement
>Reporter: Laljo John Pullokkaran
>Assignee: Jesus Camacho Rodriguez
>
> Hive supports with clause. However Hive don't cache the result set of with 
> clause and reuse it.
> Instead we inline the query definition of the with clause. This results in re 
> execution of with clause query every where it is referenced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-11049) With Clause should cache data & reuse

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez resolved HIVE-11049.

Resolution: Duplicate

Closing as duplicate of HIVE-11752. Full integration with CBO is still a 
pending task; but support for _spool_ operators needs to be added to Calcite 
first.

> With Clause should cache data & reuse 
> --
>
> Key: HIVE-11049
> URL: https://issues.apache.org/jira/browse/HIVE-11049
> Project: Hive
>  Issue Type: Improvement
>Reporter: Laljo John Pullokkaran
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.3.0
>
>
> Hive supports with clause. However Hive don't cache the result set of with 
> clause and reuse it.
> Instead we inline the query definition of the with clause. This results in re 
> execution of with clause query every where it is referenced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12044) Extend HivePreFilteringRule to cover more cases

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-12044:
---
Labels: newbie++  (was: )

> Extend HivePreFilteringRule to cover more cases
> ---
>
> Key: HIVE-12044
> URL: https://issues.apache.org/jira/browse/HIVE-12044
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>  Labels: newbie++
>
> The logic of traversing expressions (ExtractCommonOperands) is not recursive 
> and misses many of the use cases.
> Also check for comparison need to take in to account functions (ex: ((x+y)=10 
> and z=1) or ((x+y)=10 and z=2) ).
> Cc'ed [~jpullokkaran]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12044) Extend HivePreFilteringRule to cover more cases

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-12044:
---
Assignee: (was: Jesus Camacho Rodriguez)

> Extend HivePreFilteringRule to cover more cases
> ---
>
> Key: HIVE-12044
> URL: https://issues.apache.org/jira/browse/HIVE-12044
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 2.0.0
>Reporter: Jesus Camacho Rodriguez
>  Labels: newbie++
>
> The logic of traversing expressions (ExtractCommonOperands) is not recursive 
> and misses many of the use cases.
> Also check for comparison need to take in to account functions (ex: ((x+y)=10 
> and z=1) or ((x+y)=10 and z=2) ).
> Cc'ed [~jpullokkaran]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)