[GitHub] hive pull request #326: execute error with this class

[undeadwing]

GitHub user undeadwing opened a pull request:

https://github.com/apache/hive/pull/326

execute error with this class

when i use hive-jdbc to get data from hive, i got this exception
org.apache.commons.beanutils.ConversionException: Cannot assign value of 
type 'java.math.BigDecimal' to property 'valuess' of type 'java.math.BigInteger'
because the data type is decimal
so i change the type to solve it

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/undeadwing/hive master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/326.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #326


commit 1d74b3177892c984393844cf2fce924c44b92d07
Author: undeadwing 
Date:   2018-03-23T05:08:04Z

execute error with this class

when i use hive-jdbc to get data from hive, i got this exception
org.apache.commons.beanutils.ConversionException: Cannot assign value of 
type 'java.math.BigDecimal' to property 'valuess' of type 'java.math.BigInteger'
because the data type is decimal
so i change the type to solve it




---

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

[j . prasanth . j]

> On March 23, 2018, 2:27 a.m., Sergey Shelukhin wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
> > Lines 1105 (patched)
> > 
> >
> > is it possible to do this thru management endpoint that has appropriate 
> > acls and security?

I can add a management endpoint as well. I will leave the HTTP endpoint for 
secure cluster access only (maybe it can be useful if someone builds UI for 
this).


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/#review199835
---


On March 23, 2018, 2 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66236/
> ---
> 
> (Updated March 23, 2018, 2 a.m.)
> 
> 
> Review request for hive, Gopal V and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-19033
> https://issues.apache.org/jira/browse/HIVE-19033
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19033: Provide an option to purge LLAP IO cache
> 
> 
> Diffs
> -
> 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 6e6f5b92c998bb1b3055421a06155fad9b22e8e4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
>  4fbaac1db0a545e591f4b8af53b729d6a10dd679 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  2cd70b97b2dfad94fb7e05f78d268fb6aefe4e6e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  50a2411b4b14306342d451b6a62c629440a0cfba 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  b42f76184cb08be0f6db1be8e64f86593a89c372 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java
>  e88c819b2c26bbc4fef92d4030206853bdb33184 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
>  65b4d81000d1b4ac9041a51e5e8d8377efa25a71 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapIoMemoryServlet.java
>  153ab3554bbd5437aba39e980700551b11b333f9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> e5bc3c2bfa8615187eb1d1a3b80afb16a42b2c08 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  2c87bc28e51918d34c73edcdbb6874ec84db86cb 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  f7ebff2028dcde66d3345873a96ba1becc5ef59c 
> 
> 
> Diff: https://reviews.apache.org/r/66236/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/#review199835
---




llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
Lines 1105 (patched)


is it possible to do this thru management endpoint that has appropriate 
acls and security?



llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java
Line 142 (original), 142 (patched)


should be set to the final value that is modified below..


- Sergey Shelukhin


On March 23, 2018, 2 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66236/
> ---
> 
> (Updated March 23, 2018, 2 a.m.)
> 
> 
> Review request for hive, Gopal V and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-19033
> https://issues.apache.org/jira/browse/HIVE-19033
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19033: Provide an option to purge LLAP IO cache
> 
> 
> Diffs
> -
> 
>   llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
> 6e6f5b92c998bb1b3055421a06155fad9b22e8e4 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
>  4fbaac1db0a545e591f4b8af53b729d6a10dd679 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java
>  2cd70b97b2dfad94fb7e05f78d268fb6aefe4e6e 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
>  50a2411b4b14306342d451b6a62c629440a0cfba 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
>  b42f76184cb08be0f6db1be8e64f86593a89c372 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java
>  e88c819b2c26bbc4fef92d4030206853bdb33184 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
>  65b4d81000d1b4ac9041a51e5e8d8377efa25a71 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapIoMemoryServlet.java
>  153ab3554bbd5437aba39e980700551b11b333f9 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
> e5bc3c2bfa8615187eb1d1a3b80afb16a42b2c08 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
>  2c87bc28e51918d34c73edcdbb6874ec84db86cb 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
>  f7ebff2028dcde66d3345873a96ba1becc5ef59c 
> 
> 
> Diff: https://reviews.apache.org/r/66236/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Review Request 66237: HIVE-18971 add HS2 WM metrics for use in Grafana and such

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66237/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

.


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleMetrics.java
 a43b09db8c 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8d9b5a3194 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 3a2c19a3e6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java 
a52928cc7a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/QueryAllocationManager.java 
9885ce7221 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WmPoolMetrics.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 
f0e620c684 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezProgressMonitor.java
 a14cdb609a 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 
20a5947291 


Diff: https://reviews.apache.org/r/66237/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 66236: HIVE-19033: Provide an option to purge LLAP IO cache

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66236/
---

Review request for hive, Gopal V and Sergey Shelukhin.


Bugs: HIVE-19033
https://issues.apache.org/jira/browse/HIVE-19033


Repository: hive-git


Description
---

HIVE-19033: Provide an option to purge LLAP IO cache


Diffs
-

  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapIo.java 
6e6f5b92c998bb1b3055421a06155fad9b22e8e4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 4fbaac1db0a545e591f4b8af53b729d6a10dd679 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
2cd70b97b2dfad94fb7e05f78d268fb6aefe4e6e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 50a2411b4b14306342d451b6a62c629440a0cfba 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 b42f76184cb08be0f6db1be8e64f86593a89c372 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java
 e88c819b2c26bbc4fef92d4030206853bdb33184 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
 65b4d81000d1b4ac9041a51e5e8d8377efa25a71 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapIoMemoryServlet.java
 153ab3554bbd5437aba39e980700551b11b333f9 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
e5bc3c2bfa8615187eb1d1a3b80afb16a42b2c08 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 2c87bc28e51918d34c73edcdbb6874ec84db86cb 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 f7ebff2028dcde66d3345873a96ba1becc5ef59c 


Diff: https://reviews.apache.org/r/66236/diff/1/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-19033) Provide an option to purge LLAP IO cache

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-19033:


 Summary: Provide an option to purge LLAP IO cache
 Key: HIVE-19033
 URL: https://issues.apache.org/jira/browse/HIVE-19033
 Project: Hive
  Issue Type: Improvement
  Components: llap
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI 
tool to invoke the endpoint of all LLAP daemons. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66222: HIVE-18909: Metrics for results cache

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66222/
---

(Updated March 23, 2018, 1:26 a.m.)


Review request for hive and Gopal V.


Changes
---

Replace Stopwatch with System.currentTimeMillis


Repository: hive-git


Description
---

Add the following metrics related to the results cache:
  public static final String QC_MAX_SIZE = "qc_max_size";
  public static final String QC_CURRENT_SIZE = "qc_current_size";
  public static final String QC_VALID_ENTRIES = "qc_valid_entries";
  public static final String QC_LOOKUPS = "qc_lookups";
  public static final String QC_VALID_HITS = "qc_valid_hits";
  public static final String QC_PENDING_HITS = "qc_pending_hits";
  public static final String QC_PENDING_FAILS = "qc_pending_fails";
  public static final String QC_PENDING_FAILS_WAIT_TIME = 
"qc_pending_fails_wait_time";
  public static final String QC_PENDING_SUCCESS_WAIT_TIME = 
"qc_pending_success_wait_time";
  // Queries rejected from being cached due to non-deterministic functions, 
temp tables, or other conditions.
  public static final String QC_INVALID_FOR_CACHING = "qc_invalid_for_caching";
  // Queries rejected from being cached because they exceeded the max cache 
entry size.
  public static final String QC_REJECTED_TOO_LARGE = "qc_rejected_too_large";
  public static final String QC_TOTAL_ENTRIES_ADDED = "qc_total_entries_added";


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
 ecf27ceabe 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
4fa1044232 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 36f6bcd069 


Diff: https://reviews.apache.org/r/66222/diff/2/

Changes: https://reviews.apache.org/r/66222/diff/1-2/


Testing
---


Thanks,

Jason Dere

Re: Review Request 66222: HIVE-18909: Metrics for results cache

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66222/#review199821
---




ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java
Lines 277 (patched)


Guava has compat issues specifically for StopWatch - System.nanoTime is 
likely to be better than the StopWatch


- Gopal V


On March 22, 2018, 6:08 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66222/
> ---
> 
> (Updated March 22, 2018, 6:08 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the following metrics related to the results cache:
>   public static final String QC_MAX_SIZE = "qc_max_size";
>   public static final String QC_CURRENT_SIZE = "qc_current_size";
>   public static final String QC_VALID_ENTRIES = "qc_valid_entries";
>   public static final String QC_LOOKUPS = "qc_lookups";
>   public static final String QC_VALID_HITS = "qc_valid_hits";
>   public static final String QC_PENDING_HITS = "qc_pending_hits";
>   public static final String QC_PENDING_FAILS = "qc_pending_fails";
>   public static final String QC_PENDING_FAILS_WAIT_TIME = 
> "qc_pending_fails_wait_time";
>   public static final String QC_PENDING_SUCCESS_WAIT_TIME = 
> "qc_pending_success_wait_time";
>   // Queries rejected from being cached due to non-deterministic functions, 
> temp tables, or other conditions.
>   public static final String QC_INVALID_FOR_CACHING = 
> "qc_invalid_for_caching";
>   // Queries rejected from being cached because they exceeded the max cache 
> entry size.
>   public static final String QC_REJECTED_TOO_LARGE = "qc_rejected_too_large";
>   public static final String QC_TOTAL_ENTRIES_ADDED = 
> "qc_total_entries_added";
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
>  ecf27ceabe 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
> 4fa1044232 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 36f6bcd069 
> 
> 
> Diff: https://reviews.apache.org/r/66222/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

[jira] [Created] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT

[Matt McCline (JIRA)]

Matt McCline created HIVE-19032:
---

 Summary: Vectorization: Disable GROUP BY aggregations with DISTINCT
 Key: HIVE-19032
 URL: https://issues.apache.org/jira/browse/HIVE-19032
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


Vectorized GROUP BY does not support DISTINCT aggregation functions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] Should we release Hive Storage API 2.5.0 rc0?

[Owen O'Malley]

All,
   I'd like to make a new release of the storage-api with some of the new
features.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.5.0-rc0
tar ball: http://home.apache.org/~omalley/storage-2.5.0/

Thanks,
   Owen

[GitHub] hive pull request #323: HIVE-19013. Fix various storage-api build issues.

[omalley]

Github user omalley closed the pull request at:

https://github.com/apache/hive/pull/323


---

[GitHub] hive pull request #325: Hive19031 Mark duplicate configs in HiveConf as depr...

[alanfgates]

GitHub user alanfgates opened a pull request:

https://github.com/apache/hive/pull/325

Hive19031 Mark duplicate configs in HiveConf as deprecated



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alanfgates/hive hive19031

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/325.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #325


commit 7b9246825eff191cfe819b20fa10649684fa15b4
Author: Alan Gates 
Date:   2018-03-22T19:42:29Z

Cleanup and refactoring in MetastoreConf.

commit e91d26594f2010417dd6a88311644d4a393c1576
Author: Alan Gates 
Date:   2018-03-22T22:30:39Z

HIVE-19031 Mark duplicate configs in HiveConf as deprecated




---

[jira] [Created] (HIVE-19031) Mark duplicate configs in HiveConf as deprecated

[Alan Gates (JIRA)]

Alan Gates created HIVE-19031:
-

 Summary: Mark duplicate configs in HiveConf as deprecated
 Key: HIVE-19031
 URL: https://issues.apache.org/jira/browse/HIVE-19031
 Project: Hive
  Issue Type: Sub-task
  Components: Configuration, Standalone Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates


There are a number of configuration values that were copied from HiveConf to 
MetastoreConf.  They have been left in HiveConf for backwards compatibility.  
But they need to be marked as deprecated so that users know to use the new 
values in MetastoreConf.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19030) Update Wiki with new rules for Load Data

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19030:
-

 Summary: Update Wiki with new rules for Load Data
 Key: HIVE-19030
 URL: https://issues.apache.org/jira/browse/HIVE-19030
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Deepak Jaiswal


[~djaiswal] could you please update
https://cwiki.apache.org/confluence/display/hive/languagemanual+dml#LanguageManualDML-Loadingfilesintotables
with latest rules based on HIVE-18125



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66185: JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline

[Vaibhav Gumashta]

> On March 22, 2018, 4:22 p.m., Vihang Karajgaonkar wrote:
> > I am a bit confused here. If the full url can be provided in the config 
> > file by the user, how is it better than just creating a environment 
> > variable like BEELINE_URL_ and use it instead of adding it in the 
> > config file? I think the objective of this config file was to automatically 
> > figure out the connection url based on hive-site.xml and the additional 
> > beeline-hs2-connection.xml to override/augment the information from 
> > hive-site.xml
> > 
> > The current code is structured such that all keys start with 
> > beeline.hs2.connection. and components of the url are parsed automatically 
> > using the values of those keys. If we want to add full support of named 
> > urls which can have completely different url components like session vars 
> > etc, what do you think of adding a new prefix key of the form 
> > beeline.hs2.connection. and then the existing code will work exactly 
> > like it does currently but instead will parse the keys starting with 
> > beeline.hs2.connection.. For example, a named url called "blue" will 
> > be constructed using all the keys from beeline.hs2.connection.blue. That 
> > way we reuse existing logic. The beeline will be invoked like beeline -c 
> > blue. Do you see any problems with this approach? This way the user doesn't 
> > have to provide all the url components which can be reused from 
> > hive-site.xml (like the nasty ssl, kerberos settings)

Thanks for your feedback. Discussed a bit offline with Thejas as well. Let me 
add more details on the use case:
Suppose you have 2 different sets of HS2 instances running on the cluster, a 
beeline shell will only be able to parse one hive-site.xml (set 1 for example). 
To be able to connect to set 2, it would be nice to have an installer 
(something like Apache Ambari) managed beeline-site.xml, which can publish the 
named urls (and also regenerate the named urls if the admin makes any change in 
the cluster manager), which can be used by the beeline shell. Once the base 
connection url is figured out, beeline-hs2-connection.xml can then be used to 
overlay user specific driver configs like it is doing right now. Hope that 
clarifies the use case. I'll post an updated patch based on your feedback above.


- Vaibhav


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66185/#review199770
---


On March 20, 2018, 10:54 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66185/
> ---
> 
> (Updated March 20, 2018, 10:54 p.m.)
> 
> 
> Review request for hive, Thejas Nair and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18963
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 402fae 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  b769e8581f 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  f635b40633 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  2801ebee09 
>   beeline/src/main/resources/BeeLine.properties 6fca953836 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  1d17887417 
>   beeline/src/test/resources/test-hs2-named-connection-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/BeelineWithHS2ConnectionFileTestBase.java
>  3da31ad8a9 
> 
> 
> Diff: https://reviews.apache.org/r/66185/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-19029) Load Data should prevent loading acid files

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19029:
-

 Summary: Load Data should prevent loading acid files
 Key: HIVE-19029
 URL: https://issues.apache.org/jira/browse/HIVE-19029
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


{{Load Data into T}} where T is acid should check to make sure input files were 
not copied from another Acid table, i.e. make sure the files don't have Acid 
metadata columns.

AcidUtils.MetaData.isRawFormat()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19028) Insert values query gives Failed to retrieve table statistics

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-19028:
--

 Summary: Insert values query gives Failed to retrieve table 
statistics
 Key: HIVE-19028
 URL: https://issues.apache.org/jira/browse/HIVE-19028
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Thejas M Nair
Assignee: Jesus Camacho Rodriguez


INSERT INTO `VERSION` VALUES (1, '3.0.0', 'Hive release version 3.0.0') gives 
error like -

2018-03-22T17:09:42,680 ERROR [946f78ac-24e9-493b-b877-336d3006a9b5 
HiveServer2-Handler-Pool: Thread-87]: stats.StatsUtils (:()) - Failed to 
retrieve table statistics: 
org.apache.hadoop.hive.ql.metadata.HiveException: 
NoSuchObjectException(message:Specified database/table does not exist : 
_dummy_database._dummy_table)

The query proceeds, but this can be confusing to users, support team etc

Full stack is here - 
https://gist.github.com/thejasmn/7e33ad38e7bc552e3a659adc041ba200



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19027) Make materializations invalidation cache work with multiple active remote metastores

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-19027:
--

 Summary: Make materializations invalidation cache work with 
multiple active remote metastores
 Key: HIVE-19027
 URL: https://issues.apache.org/jira/browse/HIVE-19027
 Project: Hive
  Issue Type: Improvement
  Components: Materialized views
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


The main points:
 - Only MVs stored in transactional tables can have a time window value of 0. 
Those are the only MVs that can be guaranteed to not be outdated when a query 
is executed, if we use custom storage handlers to store the materialized view, 
we cannot make any promises.
 - For MVs that +cannot be outdated+, we do not check the metastore. Instead, 
comparison is based on valid write id lists.
 - For MVs that +can be outdated+, we still rely on the invalidation cache.
 ** The window for valid outdated MVs can be specified in intervals of 1 minute 
(less than that, it is difficult to have any guarantees about whether the MV is 
actually outdated by less than a minute or not).
 ** The async loading is done every interval / 2 (or probably better, we can 
make it configurable).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: PTest stuck

[Jesus Camacho Rodriguez]

There seems to be something wrong still, there are +330 failures, it seems like 
test config file is not being picked up correctly. What I have seen so far is 
that TestCliDriver is picking up q files that it should not, while 
TestMiniLlapLocalCliDriver does not seem to be running at all.

-Jesús


On 3/22/18, 8:06 AM, "Peter Vary"  wrote:

The disks was full.
Removed log files, and test runs.

Restarted ptest

Please reschedule your tests please.

Thanks,
Peter

> On Mar 22, 2018, at 9:29 AM, Prasanth Jayachandran 
 wrote:
> 
> The tests aren’t running anymore.
> 
> Thanks
> Prasanth
> 
>> On Mar 21, 2018, at 6:52 PM, Vihang Karajgaonkar  
wrote:
>> 
>> Yeah. Looks like there was something wrong since the job was running 
over 5
>> hrs. I aborted the job and updated that JIRA informing the author about 
the
>> same. Lets monitor if the issue happens with other jobs as well.
>> 
>> On Wed, Mar 21, 2018 at 5:46 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> https://builds.apache.org/view/H-L/view/Hive/job/
>>> PreCommit-HIVE-Build/9735/ seems to be stuck in git gc loop. Anyone know
>>> what might be happening? Should we kill it and see if it happens again?
>>> 
>>> Thanks
>>> Prasanth
>>> 
> 

[jira] [Created] (HIVE-19026) Configurable serde for druid kafka indexing

[Nishant Bangarwa (JIRA)]

Nishant Bangarwa created HIVE-19026:
---

 Summary: Configurable serde for druid kafka indexing 
 Key: HIVE-19026
 URL: https://issues.apache.org/jira/browse/HIVE-19026
 Project: Hive
  Issue Type: Task
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


https://issues.apache.org/jira/browse/HIVE-18976 introduces support for setting 
up druid kafka-indexing service. 
Input serialization should be configurable. for now we can say we only support 
json, but there should be a mechanism to support other formats. Perhaps, we can 
make use of Hive's serde library like LazySimpleSerde etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19025) spurious ACID logs from HS2

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-19025:
---

 Summary: spurious ACID logs from HS2
 Key: HIVE-19025
 URL: https://issues.apache.org/jira/browse/HIVE-19025
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Eugene Koifman


I'm running some version close to current master, and see the following in HS2 
stdout.
I'm running a simple select query with no errors and no special transactional 
logic. Nothing else is running. 
{noformat}
18/03/22 15:55:34 INFO client.RMProxy: Connecting to ResourceManager at [snip]
OK
Error rolling back: Can't call rollback when autocommit=true
...
18/03/22 15:56:26 INFO reducesink.VectorReduceSinkObjectHashOperator: 
VectorReduceSinkObjectHashOperator constructor vectorReduceSinkInfo 
org.apache.hadoop.hive.ql.plan.VectorReduceSinkInfo@4124cdaa
Error rolling back: Can't call rollback when autocommit=true
Query ID = sershe_20180322155619_4c58bfa4-ff93-4d4f-8a11-7ddd65c5d2c6
Total jobs = 1
Launching Job 1 out of 1
Error rolling back: Can't call rollback when autocommit=true
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19024) Vectorization: Disable complex type constants

[Matt McCline (JIRA)]

Matt McCline created HIVE-19024:
---

 Summary: Vectorization: Disable complex type constants
 Key: HIVE-19024
 URL: https://issues.apache.org/jira/browse/HIVE-19024
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Matt McCline
Assignee: Matt McCline


Currently, complex type constants are not detected and cause execution failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66224: HIVE-19021 WM counters are not properly propagated from LLAP to AM

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66224/#review199801
---


Ship it!




Ship It!

- Prasanth_J


On March 22, 2018, 7:15 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66224/
> ---
> 
> (Updated March 22, 2018, 7:15 p.m.)
> 
> 
> Review request for hive, Prasanth_J and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
>  8287adb636 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  8cd723d2e0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java
>  b05e0b9e43 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  b484a13e48 
> 
> 
> Diff: https://reviews.apache.org/r/66224/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66224: HIVE-19021 WM counters are not properly propagated from LLAP to AM

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66224/
---

(Updated March 22, 2018, 7:15 p.m.)


Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
 8287adb636 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 8cd723d2e0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java
 b05e0b9e43 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 b484a13e48 


Diff: https://reviews.apache.org/r/66224/diff/3/

Changes: https://reviews.apache.org/r/66224/diff/2-3/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66224: HIVE-19021 WM counters are not properly propagated from LLAP to AM

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66224/
---

(Updated March 22, 2018, 7:13 p.m.)


Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
 8287adb636 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 8cd723d2e0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java
 b05e0b9e43 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 b484a13e48 


Diff: https://reviews.apache.org/r/66224/diff/2/

Changes: https://reviews.apache.org/r/66224/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 66224: HIVE-19021 WM counters are not properly propagated from LLAP to AM

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66224/#review199799
---




llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
Lines 24 (patched)


unused import?



llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
Lines 116 (patched)


nit. whitespace


- Prasanth_J


On March 22, 2018, 7:05 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66224/
> ---
> 
> (Updated March 22, 2018, 7:05 p.m.)
> 
> 
> Review request for hive, Prasanth_J and Siddharth Seth.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
>  8287adb636 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  8cd723d2e0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java
>  b05e0b9e43 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
>  b484a13e48 
> 
> 
> Diff: https://reviews.apache.org/r/66224/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 66224: HIVE-19021 WM counters are not properly propagated from LLAP to AM

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66224/
---

Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/counters/WmFragmentCounters.java
 8287adb636 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 8cd723d2e0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java
 b05e0b9e43 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 b484a13e48 


Diff: https://reviews.apache.org/r/66224/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 66222: HIVE-18909: Metrics for results cache

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66222/
---

Review request for hive and Gopal V.


Repository: hive-git


Description
---

Add the following metrics related to the results cache:
  public static final String QC_MAX_SIZE = "qc_max_size";
  public static final String QC_CURRENT_SIZE = "qc_current_size";
  public static final String QC_VALID_ENTRIES = "qc_valid_entries";
  public static final String QC_LOOKUPS = "qc_lookups";
  public static final String QC_VALID_HITS = "qc_valid_hits";
  public static final String QC_PENDING_HITS = "qc_pending_hits";
  public static final String QC_PENDING_FAILS = "qc_pending_fails";
  public static final String QC_PENDING_FAILS_WAIT_TIME = 
"qc_pending_fails_wait_time";
  public static final String QC_PENDING_SUCCESS_WAIT_TIME = 
"qc_pending_success_wait_time";
  // Queries rejected from being cached due to non-deterministic functions, 
temp tables, or other conditions.
  public static final String QC_INVALID_FOR_CACHING = "qc_invalid_for_caching";
  // Queries rejected from being cached because they exceeded the max cache 
entry size.
  public static final String QC_REJECTED_TOO_LARGE = "qc_rejected_too_large";
  public static final String QC_TOTAL_ENTRIES_ADDED = "qc_total_entries_added";


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/common/MetricsConstant.java
 ecf27ceabe 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
4fa1044232 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 36f6bcd069 


Diff: https://reviews.apache.org/r/66222/diff/1/


Testing
---


Thanks,

Jason Dere

[jira] [Created] (HIVE-19023) Druid storage Handler still using old select query when the CBO fails

[slim bouguerra (JIRA)]

slim bouguerra created HIVE-19023:
-

 Summary: Druid storage Handler still using old select query when 
the CBO fails
 Key: HIVE-19023
 URL: https://issues.apache.org/jira/browse/HIVE-19023
 Project: Hive
  Issue Type: Improvement
  Components: Druid integration
Reporter: slim bouguerra
Assignee: slim bouguerra


See usage of function {code} 
org.apache.hadoop.hive.druid.io.DruidQueryBasedInputFormat#createSelectStarQuery{code}
this can be replaced by scan query that is more efficent.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #324: HIVE-19007: REPL LOAD should set the Hive configs ob...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/324

HIVE-19007: REPL LOAD should set the Hive configs obtained through WITH 
clause into the tasks created.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-19007

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/324.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #324


commit cdf529178f488472594e522cb90388aa3b4b4380
Author: Sankar Hariappan 
Date:   2018-03-21T10:37:12Z

HIVE-19007: REPL LOAD should set the Hive configs obtained through WITH 
clause into the tasks created.




---

Re: Review Request 66179: HIVE-18979

[Ashutosh Chauhan]

> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateReduceFunctionsRule.java
> > Lines 177 (patched)
> > 
> >
> > I wonder if project and aggregate transform can be generate together 
> > such that instead of generating x*x in RexNode of aggregate. Aggregate only 
> > generates x and then project computes x*x Doing that may avoid project x*x 
> > in addition to x before Gby. That will save both cpu and network.
> > Could be a follow-up.
> 
> Jesús Camacho Rodríguez wrote:
> This would need major changes in the implementation of Aggregate operator 
> itself, since the group by columns are not defined by RexNode but rather by a 
> bitset containing the input columns that we are grouping by.

Alternative would be to write rule which fires after this rule and matches on 
SEL-GBY-SEL where it removes x*x from first SEL and computes and uses it in 2nd 
SEL.


> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/groupby3.q
> > Lines 1 (patched)
> > 
> >
> > Whats the reason for this?
> 
> Jesús Camacho Rodríguez wrote:
> Skew group by (set hive.groupby.skewindata=true) does not work when there 
> are multiple DISTINCT aggregate operations. The rule might produce multiple 
> DISTINCT from a single one (e.g., AVG(DISTINCT x) decomposed into 
> SUM(DISTINCT x) and COUNT(DISTINCT x) ) and hence test fails. In non-tests 
> environment, we would just fail CBO planning. I thought this was acceptable 
> (at least for the time being), since since property is off by default. I can 
> leave a comment in the test explaining this. Please let me know what you 
> think.

I see. Turning off cbo is fine here.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66179/#review199579
---


On March 20, 2018, 7:51 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66179/
> ---
> 
> (Updated March 20, 2018, 7:51 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18979
> https://issues.apache.org/jira/browse/HIVE-18979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18979
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlVarianceAggFunction.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateReduceFunctionsRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  cb0c2b1b357c8b93c19e84e46e2793b895eff60c 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 3520d90fa8d40b2358900b58f4b45995d9c3e6fa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumEmptyIsZero.java
>  01c933c4a140f0b94614fd6e4a052cc1fb7a493f 
>   ql/src/test/queries/clientpositive/groupby3.q 
> d709d9b5ad38f3f2d6e3422d14c2bb793290c701 
>   ql/src/test/queries/clientpositive/groupby3_map_skew.q 
> f9cb46ee32a8e5639375dbb6783ffacba44b59f6 
>   ql/src/test/queries/clientpositive/udaf_binarysetfunctions_no_cbo.q 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/cbo_rp_groupby3_noskew_multi_distinct.q.out
>  bece89f0d53eb23ddc40b035f535bfe92ca87a77 
>   ql/src/test/results/clientpositive/count_dist_rewrite.q.out 
> d6ff5b75cfc562fc2befa9e2d81ab0edc44713c2 
>   ql/src/test/results/clientpositive/decimal_precision.q.out 
> 3f28106685c433607748623a9a27ef350d050327 
>   ql/src/test/results/clientpositive/decimal_udf.q.out 
> e451a186fc4a2355b8d2332de360b261bdaa4ce8 
>   ql/src/test/results/clientpositive/fetch_aggregation.q.out 
> f20320fd9b45514cb30e653887128e7e1aa59253 
>   ql/src/test/results/clientpositive/groupby3.q.out 
> 7c97174830221778a4cf9876a604655bd7d8a1e0 
>   ql/src/test/results/clientpositive/groupby3_map.q.out 
> edad22b93f3fcbb9b164a93c0a2c7055e21cec2c 
>   ql/src/test/results/clientpositive/groupby3_map_multi_distinct.q.out 
> 20344640e66a808e04f14a27b69ab49082ca9996 
>   ql/src/test/results/clientpositive/groupby3_map_skew.q.out 
> e53e62c2ce773294763b57d5eaf811dd0a94682e 
>   ql/src/test/results/clientpositive/groupby3_noskew.q.out 
> 1aa4cb6ce67d061cdbd92e5589460608e222d07f 
>   ql/src/test/results/clientpositive/groupby3_noskew_multi_distinct.q.out 
> bb964e66d60a39912c7b4933a0d9237e8b528966 
>   ql/src/test/results/clientpositive/groupby_grouping_sets3.q.out 
> e894205a27ee042bce61dfbb7f7cd75293bcca61 
>   

Re: Integration testing for clients

[Peter Vary]

I am for it, but we should be absolutely clear about them, that they will be 
changed on whim. :)

> On Mar 22, 2018, at 4:24 PM, Alan Gates  wrote:
> 
> Any reason we shouldn't publish our tests jar?  I know it isn't clean and
> nice and easy to use.  And we'd need to be explicit about any interfaces
> we're guaranteeing not to change (very few at this point I think).  But the
> tools in MetaStoreTestUtils would be really useful for other projects.
> Given that we want to encourage others to integrate with HMS this seems
> like something we should do.
> 
> Alan.
> 
> On Thu, Mar 22, 2018 at 3:49 AM, Peter Vary  wrote:
> 
>> Hi Matt,
>> 
>> The MiniHMS is an internal test utility used to test HMS features.
>> It is contained in the hive-standalone-metastore-3.0.0-SNAPSHOT-tests.jar
>> I do not think there are currently plans publish it in any of the
>> artifacts.
>> 
>> Thanks,
>> Peter
>> 
>>> On Mar 21, 2018, at 6:44 PM, Matt Burgess  wrote:
>>> 
>>> I'm looking into writing integration tests for our Hive processors in
>>> Apache NiFi to be able to test against a "real" Hive Server 2, Hive
>>> Metastore, etc.  I see in Hive 3.0 there's a MiniHMS and other test
>>> classes, but I couldn't find them in a Maven repo anywhere.  Are there
>>> artifacts published somewhere that I could use for integration
>>> testing? If not, it might be a good idea to make those available for
>>> Hive clients to test with, especially if it doesn't require a full
>>> Hive install.
>>> 
>>> Thanks,
>>> Matt
>> 
>> 

Re: Review Request 66185: JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66185/#review199770
---



I am a bit confused here. If the full url can be provided in the config file by 
the user, how is it better than just creating a environment variable like 
BEELINE_URL_ and use it instead of adding it in the config file? I think 
the objective of this config file was to automatically figure out the 
connection url based on hive-site.xml and the additional 
beeline-hs2-connection.xml to override/augment the information from 
hive-site.xml

The current code is structured such that all keys start with 
beeline.hs2.connection. and components of the url are parsed automatically 
using the values of those keys. If we want to add full support of named urls 
which can have completely different url components like session vars etc, what 
do you think of adding a new prefix key of the form 
beeline.hs2.connection. and then the existing code will work exactly like 
it does currently but instead will parse the keys starting with 
beeline.hs2.connection.. For example, a named url called "blue" will be 
constructed using all the keys from beeline.hs2.connection.blue. That way we 
reuse existing logic. The beeline will be invoked like beeline -c blue. Do you 
see any problems with this approach? This way the user doesn't have to provide 
all the url components which can be reused from hive-site.xml (like the nasty 
ssl, kerberos settings)

- Vihang Karajgaonkar


On March 20, 2018, 10:54 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66185/
> ---
> 
> (Updated March 20, 2018, 10:54 p.m.)
> 
> 
> Review request for hive, Thejas Nair and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-18963
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-18963
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 402fae 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileParser.java
>  b769e8581f 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/HS2ConnectionFileUtils.java
>  f635b40633 
>   
> beeline/src/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
>  2801ebee09 
>   beeline/src/main/resources/BeeLine.properties 6fca953836 
>   
> beeline/src/test/org/apache/hive/beeline/hs2connection/TestUserHS2ConnectionFileParser.java
>  1d17887417 
>   beeline/src/test/resources/test-hs2-named-connection-config.xml 
> PRE-CREATION 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/hs2connection/BeelineWithHS2ConnectionFileTestBase.java
>  3da31ad8a9 
> 
> 
> Diff: https://reviews.apache.org/r/66185/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

Re: Integration testing for clients

[Alan Gates]

Any reason we shouldn't publish our tests jar?  I know it isn't clean and
nice and easy to use.  And we'd need to be explicit about any interfaces
we're guaranteeing not to change (very few at this point I think).  But the
tools in MetaStoreTestUtils would be really useful for other projects.
Given that we want to encourage others to integrate with HMS this seems
like something we should do.

Alan.

On Thu, Mar 22, 2018 at 3:49 AM, Peter Vary  wrote:

> Hi Matt,
>
> The MiniHMS is an internal test utility used to test HMS features.
> It is contained in the hive-standalone-metastore-3.0.0-SNAPSHOT-tests.jar
> I do not think there are currently plans publish it in any of the
> artifacts.
>
> Thanks,
> Peter
>
> > On Mar 21, 2018, at 6:44 PM, Matt Burgess  wrote:
> >
> > I'm looking into writing integration tests for our Hive processors in
> > Apache NiFi to be able to test against a "real" Hive Server 2, Hive
> > Metastore, etc.  I see in Hive 3.0 there's a MiniHMS and other test
> > classes, but I couldn't find them in a Maven repo anywhere.  Are there
> > artifacts published somewhere that I could use for integration
> > testing? If not, it might be a good idea to make those available for
> > Hive clients to test with, especially if it doesn't require a full
> > Hive install.
> >
> > Thanks,
> > Matt
>
>

Re: PTest stuck

[Peter Vary]

The disks was full.
Removed log files, and test runs.

Restarted ptest

Please reschedule your tests please.

Thanks,
Peter

> On Mar 22, 2018, at 9:29 AM, Prasanth Jayachandran 
>  wrote:
> 
> The tests aren’t running anymore.
> 
> Thanks
> Prasanth
> 
>> On Mar 21, 2018, at 6:52 PM, Vihang Karajgaonkar  wrote:
>> 
>> Yeah. Looks like there was something wrong since the job was running over 5
>> hrs. I aborted the job and updated that JIRA informing the author about the
>> same. Lets monitor if the issue happens with other jobs as well.
>> 
>> On Wed, Mar 21, 2018 at 5:46 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> https://builds.apache.org/view/H-L/view/Hive/job/
>>> PreCommit-HIVE-Build/9735/ seems to be stuck in git gc loop. Anyone know
>>> what might be happening? Should we kill it and see if it happens again?
>>> 
>>> Thanks
>>> Prasanth
>>> 
> 

Re: Review Request 66179: HIVE-18979

[Jesús Camacho Rodríguez]

> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateReduceFunctionsRule.java
> > Lines 177 (patched)
> > 
> >
> > I wonder if project and aggregate transform can be generate together 
> > such that instead of generating x*x in RexNode of aggregate. Aggregate only 
> > generates x and then project computes x*x Doing that may avoid project x*x 
> > in addition to x before Gby. That will save both cpu and network.
> > Could be a follow-up.

This would need major changes in the implementation of Aggregate operator 
itself, since the group by columns are not defined by RexNode but rather by a 
bitset containing the input columns that we are grouping by.


> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/groupby3.q
> > Lines 1 (patched)
> > 
> >
> > Whats the reason for this?

Skew group by (set hive.groupby.skewindata=true) does not work when there are 
multiple DISTINCT aggregate operations. The rule might produce multiple 
DISTINCT from a single one (e.g., AVG(DISTINCT x) decomposed into SUM(DISTINCT 
x) and COUNT(DISTINCT x) ) and hence test fails. In non-tests environment, we 
would just fail CBO planning. I thought this was acceptable (at least for the 
time being), since since property is off by default. I can leave a comment in 
the test explaining this. Please let me know what you think.


> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/groupby3_map_skew.q
> > Lines 1 (patched)
> > 
> >
> > Whats the reason for this?

Idem as above.


> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/decimal_udf.q.out
> > Line 1852 (original), 1852 (patched)
> > 
> >
> > It will be good to avoid project key * key in addition to key, before 
> > Gby. Left a comment in Rule as well.

Left comment above, this could be done, but work will be non-trivial.


> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/udaf_binarysetfunctions.q.out
> > Line 396 (original), 396 (patched)
> > 
> >
> > Some functions didn't get decomposed?

You are right, currently rule only supports decomposition of AVG, STDDEV_POP, 
STDDEV_SAMP, VAR_POP, and VAR_SAMP.
The function that are not decomposed above are: CORR(y,x), COVAR_SAMP(y,x), 
COVAR_POP(y,x), REGR_COUNT(y,x), REGR_SLOPE(y,x), REGR_INTERCEPT(y,x), 
REGR_R2(y,x), REGR_SXX(y,x), REGR_SYY(y,x), REGR_SXY(y,x), REGR_AVGX(y,x), 
REGR_AVGY(y,x).
I can create follow-up to explore extensions for the rule with other rewritings 
(I guess at least COVAR and CORR are worth supporting).


> On March 21, 2018, 11:54 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/udaf_number_format.q.out
> > Line 75 (original), 79 (patched)
> > 
> >
> > avg('a') is ill-formed query. Ideally, Hive should throw exception for 
> > this. (Sql server and pg throws). But Mysql gives 0.0 so i guess we are 
> > making progress to being more in line with non-compliant db :)

Maybe if we all become non-compliant, we will re-define the standard... lol Now 
seriously, expression is ill-formed indeed, failure would be more logical, I 
remember we saw similar issues with casting of literals to NULL instead of 
failing. If we want to make Hive type system more strict, it could be done as 
part of that work.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66179/#review199579
---


On March 20, 2018, 7:51 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66179/
> ---
> 
> (Updated March 20, 2018, 7:51 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-18979
> https://issues.apache.org/jira/browse/HIVE-18979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18979
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/functions/HiveSqlVarianceAggFunction.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateReduceFunctionsRule.java
>  PRE-CREATION 
>   
> 

Re: Review Request 66188: HIVE-18986 Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66188/#review199751
---



Hi Aihua,

One small nit, and a question from here.

Thanks,
Peter


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Lines 154-158 (original), 154-158 (patched)


It might be a good idea, to use this around our batching as well:
- DatabaseProduct.needsInBatching(dbType)

What do you think @Aihua?



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 842 (patched)


nit: typo: "queris"


- Peter Vary


On March 21, 2018, 6:57 p.m., Aihua Xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66188/
> ---
> 
> (Updated March 21, 2018, 6:57 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Yongzhi Chen.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> If the table contains a lot of columns e.g, 5k, simple table rename would 
> fail with the following stack trace. The issue is datanucleus can't handle 
> the query with lots of colName='c1' && colName='c2' && ... .
> 
> I'm breaking the query into multiple smaller queries and then we aggregate 
> the result together.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/alter_rename_table.q 2061850540 
>   ql/src/test/results/clientpositive/alter_rename_table.q.out 732d8a28d8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/Batchable.java
>  PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  6ead20aeaf 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  88d88ed4df 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  9f822564bd 
> 
> 
> Diff: https://reviews.apache.org/r/66188/diff/2/
> 
> 
> Testing
> ---
> 
> Manual test has been done for large column of tables.
> 
> 
> Thanks,
> 
> Aihua Xu
> 
>

Re: Integration testing for clients

[Peter Vary]

Hi Matt,

The MiniHMS is an internal test utility used to test HMS features.
It is contained in the hive-standalone-metastore-3.0.0-SNAPSHOT-tests.jar
I do not think there are currently plans publish it in any of the artifacts.

Thanks,
Peter

> On Mar 21, 2018, at 6:44 PM, Matt Burgess  wrote:
> 
> I'm looking into writing integration tests for our Hive processors in
> Apache NiFi to be able to test against a "real" Hive Server 2, Hive
> Metastore, etc.  I see in Hive 3.0 there's a MiniHMS and other test
> classes, but I couldn't find them in a Maven repo anywhere.  Are there
> artifacts published somewhere that I could use for integration
> testing? If not, it might be a good idea to make those available for
> Hive clients to test with, especially if it doesn't require a full
> Hive install.
> 
> Thanks,
> Matt

Review Request 66209: HIVE-17098 Race condition in Hbase tables

[Oleksiy Sayankin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66209/
---

Review request for hive, Ashutosh Chauhan, Zoltan Haindrich, and Zoltan 
Haindrich.


Repository: hive-git


Description
---

Initial commit


Diffs
-

  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 1ef4545 


Diff: https://reviews.apache.org/r/66209/diff/1/


Testing
---


Thanks,

Oleksiy Sayankin

Re: PTest stuck

[Prasanth Jayachandran]

The tests aren’t running anymore.

Thanks
Prasanth

> On Mar 21, 2018, at 6:52 PM, Vihang Karajgaonkar  wrote:
> 
> Yeah. Looks like there was something wrong since the job was running over 5
> hrs. I aborted the job and updated that JIRA informing the author about the
> same. Lets monitor if the issue happens with other jobs as well.
> 
> On Wed, Mar 21, 2018 at 5:46 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> 
>> https://builds.apache.org/view/H-L/view/Hive/job/
>> PreCommit-HIVE-Build/9735/ seems to be stuck in git gc loop. Anyone know
>> what might be happening? Should we kill it and see if it happens again?
>> 
>> Thanks
>> Prasanth
>>