Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 5:20 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties f452341d1c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99 
  

Re: Review Request 61976: HIVE-17409 refactor LLAP ZK registry to make the ZK-registry part reusable

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61976/
---

(Updated Sept. 1, 2017, 3:16 a.m.)


Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/LlapServiceInstanceSet.java
 PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java 
70515c4ad3 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
 cc124e76ee 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceStateChangeListener.java
 92eb8bdd13 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java 
5739d72994 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/InactiveServiceInstance.java
 9f2f3b4c3b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 ebc32a155c 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
 76fc9c73a2 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 ad17144177 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
ace94759ac 
  llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
 PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
PRE-CREATION 
  llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZookeeperUtils.java 
PRE-CREATION 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
201f5fa555 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
 1b57e38b05 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
 ebc3437245 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 ff00aba110 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 6bedccbd18 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
 339f513eb4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java 2b57d906a2 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
 a5ed308da1 


Diff: https://reviews.apache.org/r/61976/diff/2/

Changes: https://reviews.apache.org/r/61976/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 61976: HIVE-17409 refactor LLAP ZK registry to make the ZK-registry part reusable

[Sergey Shelukhin]

> On Sept. 1, 2017, 12:29 a.m., Prasanth_J wrote:
> > llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
> > Lines 488 (patched)
> > 
> >
> > This can be untied. InstanceStateChangeListener can live outside of 
> > registry and can be default listener. Consumers of registry can register 
> > its own listener.

This listener translates the raw ZK events into logical instance events. I 
think it should be inside the registry.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61976/#review184328
---


On Aug. 29, 2017, 7:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61976/
> ---
> 
> (Updated Aug. 29, 2017, 7:45 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/LlapServiceInstanceSet.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java
>  70515c4ad3 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
>  cc124e76ee 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceStateChangeListener.java
>  92eb8bdd13 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5739d72994 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/InactiveServiceInstance.java
>  9f2f3b4c3b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  ebc32a155c 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  76fc9c73a2 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  ad17144177 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  ace94759ac 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> PRE-CREATION 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 201f5fa555 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
>  1b57e38b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  ebc3437245 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  ff00aba110 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  6bedccbd18 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
>  339f513eb4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java 2b57d906a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a5ed308da1 
> 
> 
> Diff: https://reviews.apache.org/r/61976/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[GitHub] hive pull request #241: HIVE 17367: IMPORT table doesn't load from data dump...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/241


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-17428) REPL LOAD of ALTER_PARTITION event doesn't create import tasks if the partition doesn't exist during analyze phase.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-17428:
---

 Summary: REPL LOAD of ALTER_PARTITION event doesn't create import 
tasks if the partition doesn't exist during analyze phase.
 Key: HIVE-17428
 URL: https://issues.apache.org/jira/browse/HIVE-17428
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, repl
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


If the incremental dump event sequence have ADD_PARTITION followed by 
ALTER_PARTITION doesn't create any task for ALTER_PARTITION event as the 
partition doesn't exist during analyze phase. Due to this REPL STATUS returns 
wrong last repl ID.
Scenario:
1. Create DB
2. Create partitioned table.
3. Bootstrap dump and load
4. Insert into table to a dynamically created partition.
- This insert generate ADD_PARTITION and ALTER_PARTITION events.
5. Incremental dump and load.
- Load will be successful.
- But the last repl ID set was incorrect as ALTER_PARTITION event was never 
applied.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61867: HIVE-17297 allow AM to use LLAP guaranteed tasks (excludes generated code)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61867/
---

(Updated Sept. 1, 2017, 1:42 a.m.)


Review request for hive and Zhiyuan Yang.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  
llap-client/src/java/org/apache/hadoop/hive/llap/tez/LlapProtocolClientProxy.java
 fe59ab58b7 
  llap-common/pom.xml bb40996a47 
  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java aa69752cc6 
  
llap-common/src/java/org/apache/hadoop/hive/llap/impl/LlapPluginProtocolClientImpl.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/impl/ProtobufProxy.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/protocol/LlapPluginProtocolPB.java
 PRE-CREATION 
  llap-common/src/protobuf/LlapPluginProtocol.proto PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 872a2493ab 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 ff00aba110 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 6bedccbd18 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/endpoint/LlapPluginPolicyProvider.java
 PRE-CREATION 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/endpoint/LlapPluginSecurityInfo.java
 PRE-CREATION 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/endpoint/LlapPluginServerImpl.java
 PRE-CREATION 
  
llap-tez/src/main/resources/META-INF/services/org.apache.hadoop.security.SecurityInfo
 PRE-CREATION 
  
llap-tez/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
 PRE-CREATION 
  
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
 339f513eb4 


Diff: https://reviews.apache.org/r/61867/diff/2/

Changes: https://reviews.apache.org/r/61867/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-17427) Semijoin Hint : If a hint is not used, give a warning

[Deepak Jaiswal (JIRA)]

Deepak Jaiswal created HIVE-17427:
-

 Summary: Semijoin Hint : If a hint is not used, give a warning
 Key: HIVE-17427
 URL: https://issues.apache.org/jira/browse/HIVE-17427
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


If a hint is not used, give a warning to the user.
This could happen due to several factors, operator cycles, task cycles or 
simply a typo.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60289: HIVE-15665 LLAP: OrcFileMetadata objects in cache can impact heap usage

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60289/
---

(Updated Sept. 1, 2017, 12:41 a.m.)


Review request for hive, Gopal V and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
c5248ceb5f 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
f42622b892 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 6edd84b8b0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 b5db3029d1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 dc053ee7cf 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 b9d7a77d5b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcMetadataCache.java
 601b622b49 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 4565d11988 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ParquetMetadataCacheImpl.java
 b61a8ca022 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 13c7767a3b 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 03a955c6f7 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 69a9f9f35e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReader.java 
7540e72b53 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
690cce798e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java cdd58df370 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
d47ba6b31a 
  ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out 8af84dce19 
  ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out 4536cbbfb9 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 77b7f5a2f7 
  ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out 
b799527e30 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
403c3ada61 


Diff: https://reviews.apache.org/r/60289/diff/4/

Changes: https://reviews.apache.org/r/60289/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 61976: HIVE-17409 refactor LLAP ZK registry to make the ZK-registry part reusable

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61976/#review184328
---




llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 220 (patched)


nit: can be moved to util class for reuse.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 244 (patched)


Does it have to be llap zk user? Can this config be renamed generically?



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 488 (patched)


This can be untied. InstanceStateChangeListener can live outside of 
registry and can be default listener. Consumers of registry can register its 
own listener.



llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java
Lines 595 (patched)


This can be separate class too outside of registry for reuse.


- Prasanth_J


On Aug. 29, 2017, 7:45 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61976/
> ---
> 
> (Updated Aug. 29, 2017, 7:45 p.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/LlapServiceInstanceSet.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstance.java
>  70515c4ad3 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
>  cc124e76ee 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceStateChangeListener.java
>  92eb8bdd13 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceRegistry.java
>  5739d72994 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/InactiveServiceInstance.java
>  9f2f3b4c3b 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
>  ebc32a155c 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapRegistryService.java
>  76fc9c73a2 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
>  ad17144177 
>   
> llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java
>  ace94759ac 
>   llap-client/src/java/org/apache/hadoop/hive/registry/ServiceInstance.java 
> PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ServiceInstanceBase.java
>  PRE-CREATION 
>   
> llap-client/src/java/org/apache/hadoop/hive/registry/impl/ZkRegistryBase.java 
> PRE-CREATION 
>   
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
> 201f5fa555 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
>  1b57e38b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/services/impl/LlapWebServices.java
>  ebc3437245 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
>  ff00aba110 
>   
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
>  6bedccbd18 
>   
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
>  339f513eb4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java 2b57d906a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapClusterStateForCompile.java
>  a5ed308da1 
> 
> 
> Diff: https://reviews.apache.org/r/61976/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

[Vineet Garg]

> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 1672 (patched)
> > 
> >
> > varchar has length in its type. We shall use that.

Can you elobrate please? Where is varchar type? I don't see length field in 
VarcharTypeInfo.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 199 (patched)
> > 
> >
> > Need to pass shouldEstimateStats to this function so that it respects 
> > it.

Updated the code to return early if shouldEstimateStats is false.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
> > Lines 231 (patched)
> > 
> >
> > if config is false, it will return 1 as numrows, is that what we want?
> > It seems like caller has no way to figure out that stats are not 
> > estimated so it shall turn off stats based optimization.

Updated the code to return whatever we retrieve from metastore if 
shouldEstimateStat is off.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/insert1.q.out
> > Line 46 (original), 46 (patched)
> > 
> >
> > Is change to COMPLETE expected?

Yes Basic stat's state is changed to COMPLETE since now we have both data size 
and number of rows.


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/jdbc_handler.q.out
> > Line 129 (original), 129 (patched)
> > 
> >
> > Expected?

Yes


> On Aug. 29, 2017, 9:26 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
> > Line 231 (original), 231 (patched)
> > 
> >
> > Overflow in data size calculation?

Yeh this was an existing BUG in GroupBy's stats computation for aggregation 
columns. I have fixed it. Thanks for pointing it out.


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184071
---


On Sept. 1, 2017, 12:04 a.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Sept. 1, 2017, 12:04 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Sept. 1, 2017, 12:04 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99 
  

[jira] [Created] (HIVE-17426) Execution framework in hive to run tasks in parallel other than MR Tasks

[anishek (JIRA)]

anishek created HIVE-17426:
--

 Summary: Execution framework in hive to run tasks in parallel 
other than MR Tasks
 Key: HIVE-17426
 URL: https://issues.apache.org/jira/browse/HIVE-17426
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: anishek
Assignee: anishek
 Fix For: 3.0.0


the execution framework currently only runs MR Tasks in parallel when {{set 
hive.exec.parallel=true}}.

Allow other types of tasks to run in parallel as well to support replication 
scenarios in hive. TezTask / SparkTask will still not be allowed to run in 
parallel.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/#review184318
---



Latest patch doesn't address all review comments. Shall I wait for new patch 
before reviewing this?

- Ashutosh Chauhan


On Aug. 31, 2017, 10:02 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61165/
> ---
> 
> (Updated Aug. 31, 2017, 10:02 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-16811
> https://issues.apache.org/jira/browse/HIVE-16811
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch introduces estimation of statistics if stats doesn't already exist.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
>   itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
> 22790de209 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  ad29d65abb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
>   ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
>   ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
>   ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
> 156be41502 
>   ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
>   ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
>   ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
>   ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
> 23f5fcfc76 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> a7c9b3fc41 
>   ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
>   ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
>   ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
>   ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
> f77c75c232 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
> f04519f8c5 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
> f15d536945 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
> 1ec5dc415c 
>   ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
>   ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
>   ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
> 97f8d6b4cc 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 
> 8bc280cedd 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
> 4b646dcba5 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
> ab94c37fed 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
> 7e3fe4c862 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
> 054cf1af7e 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
> 9c37848896 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
> 21f1b3faff 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
> 03ff5a6659 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
> 42e4cb7a5c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
> 64c10ce953 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
> 96854a9665 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
> b77a879b78 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
> 9e6053675b 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
> 13d3accf4c 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
> 7fc90fd19f 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
> 278c55b3e4 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 
> 649afb502a 
>   ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out 
> bac4ae086c 
>   

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 31, 2017, 10:02 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99 
  

Re: Review Request 61165: HIVE-16811 Estimate statistics in absence of stats

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61165/
---

(Updated Aug. 31, 2017, 9:58 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16811
https://issues.apache.org/jira/browse/HIVE-16811


Repository: hive-git


Description
---

This patch introduces estimation of statistics if stats doesn't already exist.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e4b09a2cdd 
  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
22790de209 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 ad29d65abb 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java f2d2e2dc0b 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 487a823d92 
  ql/src/test/queries/clientpositive/join_reordering_no_stats.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out e22c3ef0fc 
  ql/src/test/results/clientpositive/annotate_stats_groupby.q.out fccfabd5d1 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 866d30a8ea 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out e3f08ea555 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out efc3c1f123 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 
156be41502 
  ql/src/test/results/clientpositive/auto_join_stats.q.out e80af96fcb 
  ql/src/test/results/clientpositive/auto_join_stats2.q.out 6ea5afa920 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out d129807f55 
  ql/src/test/results/clientpositive/cbo_rp_annotate_stats_groupby.q.out 
23f5fcfc76 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
a7c9b3fc41 
  ql/src/test/results/clientpositive/explain_rearrange.q.out 2e5f9e4626 
  ql/src/test/results/clientpositive/join19.q.out dd167ecabf 
  ql/src/test/results/clientpositive/join42.q.out 0cd31c6d66 
  ql/src/test/results/clientpositive/join43.q.out 24168ca4fe 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 
f77c75c232 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual2.q.out 
f04519f8c5 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 
f15d536945 
  ql/src/test/results/clientpositive/join_cond_pushdown_unqual4.q.out 
1ec5dc415c 
  ql/src/test/results/clientpositive/join_hive_626.q.out 33a5fbd628 
  ql/src/test/results/clientpositive/join_star.q.out 261fb53bf9 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 97f8d6b4cc 
  ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out 8bc280cedd 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_1.q.out 
4b646dcba5 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_10.q.out 
ab94c37fed 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_11.q.out 
7e3fe4c862 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_12.q.out 
054cf1af7e 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out 
9c37848896 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_14.q.out 
21f1b3faff 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_15.q.out 
03ff5a6659 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_2.q.out 
42e4cb7a5c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_3.q.out 
64c10ce953 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_4.q.out 
96854a9665 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_5.q.out 
b77a879b78 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out 
9e6053675b 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_7.q.out 
13d3accf4c 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_8.q.out 
7fc90fd19f 
  ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out 
278c55b3e4 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out 5913768a6d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out 649afb502a 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out bac4ae086c 
  ql/src/test/results/clientpositive/llap/bucketmapjoin1.q.out feb3bbcfc4 
  ql/src/test/results/clientpositive/llap/bucketmapjoin2.q.out bda8070ba5 
  ql/src/test/results/clientpositive/llap/bucketmapjoin3.q.out ae17c1b707 
  ql/src/test/results/clientpositive/llap/bucketmapjoin4.q.out aab43431e6 
  ql/src/test/results/clientpositive/llap/bucketmapjoin7.q.out 3a2cbb2b14 
  ql/src/test/results/clientpositive/llap/bucketpruning1.q.out 922f23a29e 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_2.q.out 
e4fed1124b 
  ql/src/test/results/clientpositive/llap/bucketsortoptimize_insert_6.q.out 
67e925ad99 
  

[jira] [Created] (HIVE-17425) Change MetastoreConf.ConfVars internal members to be private

[Alan Gates (JIRA)]

Alan Gates created HIVE-17425:
-

 Summary: Change MetastoreConf.ConfVars internal members to be 
private
 Key: HIVE-17425
 URL: https://issues.apache.org/jira/browse/HIVE-17425
 Project: Hive
  Issue Type: Task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates


MetastoreConf's dual use of metastore keys and Hive keys is causing confusion 
for developers.  We should make the relevant members private and provide getter 
methods with comments on when it is appropriate to use them.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17424) LLAP Parquet caching v2: IO elevator support

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-17424:
---

 Summary: LLAP Parquet caching v2: IO elevator support
 Key: HIVE-17424
 URL: https://issues.apache.org/jira/browse/HIVE-17424
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


It would be nice to move the reading, decompression etc. onto the IO 
threadpool. That would require some refactoring in Parquet codebase, or a 
considerable amount of duplicate logic. ORC path has a little bit of both - a 
lot is reused from ORC project but some logic (not code) to read the file is 
duplicated. Not sure how practical/acceptable to the community the refactoring 
might be in Parquet.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17423) LLAP Parquet caching - support file ID in splits

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-17423:
---

 Summary: LLAP Parquet caching - support file ID in splits
 Key: HIVE-17423
 URL: https://issues.apache.org/jira/browse/HIVE-17423
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


To get LLAP cache data one needs a file ID which is either an HDFS inode ID, or 
a composite of path, modification time and size. These can be embedded into 
splits for ORC, cause in particular for the former it's possible to get the IDs 
as a part of a normal file enumeration that split generation performs anyway.
If they are missing, the IDs need to be obtained for every file on the fragment 
side.
We should explore adding file IDs to Parquet splits when the cache is enabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17422) Don't dump non-native/temporary tables during incremental dump

[Tao Li (JIRA)]

Tao Li created HIVE-17422:
-

 Summary: Don't dump non-native/temporary tables during incremental 
dump
 Key: HIVE-17422
 URL: https://issues.apache.org/jira/browse/HIVE-17422
 Project: Hive
  Issue Type: Improvement
  Components: repl
Reporter: Tao Li


Currently during incremental dump, the non-native/temporary table info is 
partially dumped in metadata file and will be ignored later by the repl load. 
We can optimize it by moving the check (whether the table should be exported or 
not) earlier so that we don't save any info to dump file for such types of 
tables. CreateTableHandler already has this optimization, so we just need to 
apply similar logic to other scenarios.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17421) Clear incorrect stats after replication

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-17421:
-

 Summary: Clear incorrect stats after replication
 Key: HIVE-17421
 URL: https://issues.apache.org/jira/browse/HIVE-17421
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Daniel Dai
Assignee: Daniel Dai


After replication, some stats summary are incorrect. If 
hive.compute.query.using.stats set to true, we will get wrong result on the 
destination side.

This will not happen with bootstrap replication. This is because stats summary 
are in table properties and will be replicated to the destination. However, in 
incremental replication, this won't work. When creating table, the stats 
summary are empty (eg, numRows=0). Later when we insert data, stats summary are 
updated with update_table_column_statistics/update_partition_column_statistics, 
however, both events are not captured in incremental replication. Thus on the 
destination side, we will get count(*)=0. The simple solution is to remove 
COLUMN_STATS_ACCURATE property after incremental replication.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] hive pull request #242: HIVE-17420 : bootstrap - get replid before object du...

[anishek]

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/242

HIVE-17420 : bootstrap - get replid before object dump



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-17420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #242


commit bc5595b8a9e5ce9ce9771b11478a32c476449d5a
Author: Anishek Agarwal 
Date:   2017-08-31T19:36:00Z

HIVE-17420 : bootstrap - get replid before object dump




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #231: HIVE-17100: Improve HS2 operation logs for REPL comm...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/231


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #241: HIVE 17367

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/241

HIVE 17367



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-17367

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #241


commit 7693b9594d718388403fcf145bbca24469c712c1
Author: Sankar Hariappan 
Date:   2017-08-22T16:43:22Z

HIVE-17367: IMPORT table doesn't load from data dump if a metadata-only 
dump was already imported.

commit 18decf7565769d5f5af2c07eb950034883f09a1b
Author: Sankar Hariappan 
Date:   2017-08-25T18:53:32Z

Handled the failure case of import where metastore object is updated with 
last repl id but data copy fails




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-17420) bootstrap - get replid before object dump

[anishek (JIRA)]

anishek created HIVE-17420:
--

 Summary: bootstrap - get replid before object dump
 Key: HIVE-17420
 URL: https://issues.apache.org/jira/browse/HIVE-17420
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: anishek
 Fix For: 3.0.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62004: HIVE-17415

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62004/
---

(Updated Aug. 31, 2017, 5:42 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

Added the test to testconfiguration


Bugs: hive-17415
https://issues.apache.org/jira/browse/hive-17415


Repository: hive-git


Description
---

Hit error "SemanticException View xxx is corresponding to LIMIT, rather than a 
SelectOperator." in Hive queries when a user creates a view with limits


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties fa6a2aaea0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e8acabefa5 
  ql/src/test/queries/clientpositive/authorization_view_8.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/authorization_view_8.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/62004/diff/2/

Changes: https://reviews.apache.org/r/62004/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-17419) ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS command shows computed stats for masked tables

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-17419:
--

 Summary: ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS command 
shows computed stats for masked tables
 Key: HIVE-17419
 URL: https://issues.apache.org/jira/browse/HIVE-17419
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez


As {{ANALYZE TABLE...COMPUTE STATISTICS FOR COLUMNS}} is rewritten internally 
as a {{SELECT}} query, there is an interaction between the rewriting to mask 
the columns/rows interacts and the ColumnStatsSemanticAnalyzer rewriting that 
leads to showing the computed stats after running the command.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Read Parquet file in Hive and apply partition to External table

[Chetan Khatri]

Hello Hive Dev,

I have one question that, I have 1 parquet file now i want to do count and
apply partition, how could i do that.

Thanks.

Re: Review Request 61985: HIVE-17399

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61985/
---

(Updated Aug. 31, 2017, 7:26 a.m.)


Review request for hive, Gopal V and Jason Dere.


Changes
---

Fixed the failing test.
Use try catch block to handle NPE.
Return after assert.


Bugs: HIVE-17399
https://issues.apache.org/jira/browse/HIVE-17399


Repository: hive-git


Description
---

Do not remove semijoin branch if it feeds to TS->DPP_EVENT


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemiJoinBranchInfo.java 
5d7b9e5c6d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 1671773d4a 
  ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q b22890bc9d 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
3bd35bf2d8 


Diff: https://reviews.apache.org/r/61985/diff/3/

Changes: https://reviews.apache.org/r/61985/diff/2-3/


Testing
---


Thanks,

Deepak Jaiswal