Re: Does MERGE feature works only on TEZ in Hive-2.3?

2018-04-19 Thread Oleksiy S 
Thanks!

On Wed, Apr 18, 2018 at 7:45 PM, Eugene Koifman 
wrote:

> It works on MR as well.
>
> From: Oleksiy S 
> Reply-To: "u...@hive.apache.org" 
> Date: Wednesday, April 18, 2018 at 5:24 AM
> To: "u...@hive.apache.org" , "dev@hive.apache.org" <
> dev@hive.apache.org>
> Subject: Does MERGE feature works only on TEZ in Hive-2.3?
>
> Hi all!
>
> Does MERGE feature  works only on TEZ in Hive-2.3?
>
> see
>
>   *   https://issues.apache.org/jira/browse/HIVE-10924
>   *   https://community.hortonworks.com/articles/97113/hive-acid-
> merge-by-example.html
>
> --
> Oleksiy
>



-- 
Oleksiy

[jira] [Created] (HIVE-19245) Add Apache license to TestSparkPlan.java

2018-04-19 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-19245:
---

 Summary: Add Apache license to TestSparkPlan.java
 Key: HIVE-19245
 URL: https://issues.apache.org/jira/browse/HIVE-19245
 Project: Hive
  Issue Type: Bug
Reporter: Sahil Takiar
Assignee: Sahil Takiar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19243) Upgrade hadoop.version to 3.1.0

2018-04-19 Thread Gour Saha (JIRA) 
Gour Saha created HIVE-19243:


 Summary: Upgrade hadoop.version to 3.1.0
 Key: HIVE-19243
 URL: https://issues.apache.org/jira/browse/HIVE-19243
 Project: Hive
  Issue Type: Sub-task
Reporter: Gour Saha


Given that Hadoop 3.1.0 has been released, we need to upgrade hadoop.version to 
3.1.0. This change is required for HIVE-18037 since it depends on YARN Service 
which had its first release in 3.1.0 (and is non-existent in 3.0.0).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19254) NumberFormatException in MetaStoreUtils.isFastStatsSame

2018-04-19 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-19254:
--

 Summary: NumberFormatException in MetaStoreUtils.isFastStatsSame
 Key: HIVE-19254
 URL: https://issues.apache.org/jira/browse/HIVE-19254
 Project: Hive
  Issue Type: Bug
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


I see the following exception under some cases in the logs. This possibly 
happens when you try to add empty partitions.

{noformat}
2018-04-19T19:32:19,260 ERROR [pool-7-thread-7] metastore.RetryingHMSHandler: 
MetaException(message:java.lang.NumberFormatException: null)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6824)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:4864)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions(HiveMetaStore.java:4801)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy24.alter_partitions(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:16046)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions.getResult(ThriftHiveMetastore.java:16030)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: null
at java.lang.Long.parseLong(Long.java:552)
at java.lang.Long.parseLong(Long.java:631)
at 
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.isFastStatsSame(MetaStoreUtils.java:632)
at 
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:743)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:4827)
... 21 more
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19253) HMS ignores tableType property for external tables

2018-04-19 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-19253:
-

 Summary: HMS ignores tableType property for external tables
 Key: HIVE-19253
 URL: https://issues.apache.org/jira/browse/HIVE-19253
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.0.2, 3.0.0, 3.1.0
Reporter: Alexander Kolbasov
Assignee: Alexander Kolbasov


When someone creates a table using Thrift API they may think that setting 
tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their 
table is gone later because HMS will silently change it to managed table.

here is the offending code:

{code:java}
  private MTable convertToMTable(Table tbl) throws InvalidObjectException,
  MetaException {
...
// If the table has property EXTERNAL set, update table type
// accordingly
String tableType = tbl.getTableType();
boolean isExternal = 
Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
  if (isExternal) {
tableType = TableType.EXTERNAL_TABLE.toString();
  }
}
if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
  if (!isExternal) { // Here!
tableType = TableType.MANAGED_TABLE.toString();
  }
}
{code}

So if the EXTERNAL parameter is not set, table type is changed to managed even 
if it was external in the first place - which is wrong.

More over, in other places code looks at the table property to decide table 
type and some places look at parameter. HMS should really make its mind which 
one to use.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19255) Hive doesn't support column list specification in INSERT into statements with distribute by/Cluster by

2018-04-19 Thread Riju Trivedi (JIRA) 
Riju Trivedi created HIVE-19255:
---

 Summary: Hive doesn't support column list specification in INSERT 
into statements with distribute by/Cluster by 
 Key: HIVE-19255
 URL: https://issues.apache.org/jira/browse/HIVE-19255
 Project: Hive
  Issue Type: Bug
  Components: Parser, Query Processor, SQL
Affects Versions: 1.2.0
Reporter: Riju Trivedi


INSERT into TABLE target_table_2 partition (col3) (col1, col2,col3)

SELECT col1,col2,col3
FROM source_table 
DISTRIBUTE BY col1 
SORT BY col1,col2;

This Insert statement throws 

Error: Error while compiling statement: FAILED: SemanticException [Error 
10004]: Line 4:14 Invalid table alias or column reference 'col1':

Query is executed successfully with below workaround:

INSERT into TABLE target_table_2 partition (col3) (col1, col2,col3)
select * From (SELECT col1, col2,col3 
FROM source_table 
DISTRIBUTE BY col1 
SORT BY col1,col2) a;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66729: HIVE-19214: High throughput ingest ORC format

2018-04-19 Thread j . prasanth . j 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66729/
---

Review request for hive and Gopal V.


Bugs: HIVE-19214
https://issues.apache.org/jira/browse/HIVE-19214


Repository: hive-git


Description
---

HIVE-19214: High throughput ingest ORC format


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
73492ff99c0aad6f6aade4fb04cd80a9c982b979 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 9e8ae103ee7de59c368eb76fd49a33b866bd30fe 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 afb8fc51f22f768e3a465cda65b4ea9fcb635949 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 3d9e99cfae8fc7b658582277c9a630f7a801953d 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
fe109d7b9668f686b999df37c60016a93a251d22 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 
d850062377d182e33a6191268d50d0008d7c77de 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
4e17394653caf51cb29b4017656518a5facab8ee 
  ql/src/test/queries/clientpositive/orc_ppd_exception.q 
1513d911201c30130d611f49d1877cdacf2892fc 
  ql/src/test/queries/clientpositive/vector_acid3.q 
d4313f4ba5a8358ccf6a266564777823047640e9 
  ql/src/test/results/clientpositive/llap/vector_acid3.q.out 
46c82fcbb0d8dfd5889a4e258ba8af2670d33373 
  ql/src/test/results/clientpositive/orc_ppd_exception.q.out 
e03eb71068ebd80cc8e7453bd6129d10b18336c3 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 
6f63bfb43e5dbe4c9529dfc80787a95ba6524c01 


Diff: https://reviews.apache.org/r/66729/diff/1/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 66663: HIVE-19171 Persist runtime statistics in metastore

2018-04-19 Thread Zoltan Haindrich 


> On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java
> > Lines 97 (patched)
> > 
> >
> > This shouldn't be part of cache. Cache is size limited which expires 
> > oldest entry to keep constant size. Guava cache has those features.

this was here also because all eviction was controlled from hs2 side ; now 
there is a cleaner there - so this is not needed


> On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 11630 (patched)
> > 
> >
> > This shouldn't be an api which MSC invokes. This needs to be a thread 
> > which runs in Metastore. We already have bunch of these cleaner tasks in 
> > Metastore. Config metastore.task.threads.always has list of such cleaner 
> > threads where this should be added. MaterializationsCacheCleanerTask is an 
> > example of one such cleaner thread.

added cleaner thread


> On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql
> > Lines 687 (patched)
> > 
> >
> > Is it an unused field for future? How do you envision it being used?

this is being used; because an entry has the stats for a whole plan; the weight 
is the number of operators it contains information for;
the limit at both hs2/metastore side is for per operator stat entries; and not 
"query" count


> On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/sql/derby/hive-schema-3.0.0.derby.sql
> > Lines 688 (patched)
> > 
> >
> > Each entry should store its create time which can then be used to purge 
> > it later. See e.g., Notification_log tables which stores creation_time and 
> > uses it later to purge rows.

there is a create time :)
however it's missing from this sql file...


> On April 17, 2018, 11:47 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/thrift/hive_metastore.thrift
> > Lines 2180 (patched)
> > 
> >
> > There shouldn't be any need to pass maxRetained and maxRetainSecs 
> > params. This call shouldn't do any cleanup. Thats the job of cleaner thread 
> > which should read this params from config.

ok


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3/#review201359
---


On April 17, 2018, 12:49 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3/
> ---
> 
> (Updated April 17, 2018, 12:49 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-19171
> https://issues.apache.org/jira/browse/HIVE-19171
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * wip patch
> * also contains HIVE-19009
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> e540d023bdda5fbb6152d7ba93c134c0542bf9dd 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7aca288fddcb974603190f342ae9d1b3a71 
>   itests/src/test/resources/testconfiguration.properties 
> 48d62a8bf9bcffc2723acb864973a9a53172dceb 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 3cdad284efe92b38f84809ec1a4034aee408ecd8 
>   metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> 9ca8b0007e4cbd0c030a37aa35155526d96bb616 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> a88453c97835db847d74b4b4c3ef318d4d6c0ce5 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> d3fbf07de4613a35f2d172d40053bb26de01407d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
> e87bbceb7aa2a1d0a1f1cc1f00ece2990b9bfd34 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java
>  c3dc848a32a39c30981a0a95678eed01ba547856 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java 
> e15a49f8380a551f69e11d6b0c560116ffc4a83a 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 
>

Re: Review Request 66663: HIVE-19171 Persist runtime statistics in metastore

2018-04-19 Thread Zoltan Haindrich 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3/
---

(Updated April 19, 2018, 3:54 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

update to wip#03 patch


Bugs: HIVE-19171
https://issues.apache.org/jira/browse/HIVE-19171


Repository: hive-git


Description
---

* wip patch
* also contains HIVE-19009


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff99c 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7aca2 
  itests/src/test/resources/testconfiguration.properties d26f0ccb17 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java fad8c0ffe9 
  metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 9ca8b0007e 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java f761fffbf1 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
e15c5b7b66 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
e87bbceb7a 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java 
c3dc848a32 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java e15a49f838 
  ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java a61a47e390 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 5b7f4c3560 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 54b705db6e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
72092ce4ae 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java a37280407d 
  
ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java 
b5a3c2459f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java df5aa0c679 
  ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java 
2b0d23c6f2 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java 8a5595d210 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java 
4ee3c14b39 
  ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java f731315956 
  ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java 52e18a8030 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java
 0afc533da8 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestRuntimeStatsPersistence.java
 PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java 
18aeb3338b 
  ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java 
b7263005ed 
  ql/src/test/queries/clientpositive/runtime_stats_hs2.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/runtime_stats_hs2.q.out PRE-CREATION 
  standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 802d8e3fb2 
  standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
dfa13a0614 
  
standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp
 c0a39f80e0 
  standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 2c95007daa 
  standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 
99024279c5 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RuntimeStat.java
 PRE-CREATION 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 a354f27cad 
  standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php 
9c949429c5 
  standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php c4969d567f 
  
standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 079c7fc322 
  
standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 d241414bc3 
  standalone-metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py 
9bf9843314 
  standalone-metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb 3dbe4d8068 
  standalone-metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 
58ebd29523 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 ae9ec5cad8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 7ba053d3e8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 27f8775a10 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 a7acdcbc23

Re: Review Request 65745: HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS workaround is buggy.

2018-04-19 Thread Alexander Kolbasov 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65745/
---

(Updated April 19, 2018, 3:51 p.m.)


Review request for hive, Andrew Sherman, Janaki Lahorani, Zoltan Haindrich, 
Sahil Takiar, Thejas Nair, and Vihang Karajgaonkar.


Changes
---

Merged with latest master.


Summary (updated)
-

HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS 
workaround is buggy.


Bugs: HIVE-18743
https://issues.apache.org/jira/browse/HIVE-18743


Repository: hive-git


Description (updated)
---

HIVE-18743: CREATE TABLE on S3 data can be extremely slow. DO_NOT_UPDATE_STATS 
workaround is buggy.


Diffs (updated)
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 60bed9841f65fd6ef74a14be3f2723c1825c7adc 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 ae9ec5cad812d49ee30ebb52e0dba5c0325ca78e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 d022bc0343901a588722b49d476a5eb6ac1f8104 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
 d6c13d3f2a4dc54dab978f891b51d623a4d29762 


Diff: https://reviews.apache.org/r/65745/diff/8/

Changes: https://reviews.apache.org/r/65745/diff/7-8/


Testing
---

Added both positive unit test verifying that stats are updated and negative 
test verifying that stats are not updated when they shouldn't be.


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-19246) Update golden files for negative tests

2018-04-19 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19246:
---

 Summary: Update golden files for negative tests
 Key: HIVE-19246
 URL: https://issues.apache.org/jira/browse/HIVE-19246
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


+Error during job, obtaining debugging information...

shows up in q.out due to one of recent changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19244) Set read ACL for /apache_atlas/active_server_info znode in HA for Knox proxy to read

2018-04-19 Thread Nixon Rodrigues (JIRA) 
Nixon Rodrigues created HIVE-19244:
--

 Summary: Set read ACL for /apache_atlas/active_server_info znode 
in HA for Knox proxy to read
 Key: HIVE-19244
 URL: https://issues.apache.org/jira/browse/HIVE-19244
 Project: Hive
  Issue Type: Bug
Reporter: Nixon Rodrigues






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66131: HIVE-18983: Add support for table properties inheritance in Create table like

2018-04-19 Thread Igor Kryvenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66131/
---

(Updated Апрель 19, 2018, 12:53 п.п.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and 
Prasanth_J.


Bugs: HIVE-18983
https://issues.apache.org/jira/browse/HIVE-18983


Repository: hive-git


Description
---

Currently, Create table like support table properties.
But it doesn't inherit table properties from original table.
It uses just default table properties and doesn't inherit properties from 
original table.
It would be great if create table like will inherit origin table properties and 
will override them if they are specified in query .


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 88b5ed8 
  ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
e74be62 
  ql/src/test/results/clientpositive/create_like.q.out 7c26a59 
  ql/src/test/results/clientpositive/create_like2.q.out 601b929 
  ql/src/test/results/clientpositive/explain_ddl.q.out 6506e89 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
 78ea01d 


Diff: https://reviews.apache.org/r/66131/diff/4/

Changes: https://reviews.apache.org/r/66131/diff/3-4/


Testing
---


Thanks,

Igor Kryvenko

[jira] [Created] (HIVE-19248) Hive replication cause file copy failures if HDFS block size differs across clusters

2018-04-19 Thread Sankar Hariappan (JIRA) 
Sankar Hariappan created HIVE-19248:
---

 Summary: Hive replication cause file copy failures if HDFS block 
size differs across clusters
 Key: HIVE-19248
 URL: https://issues.apache.org/jira/browse/HIVE-19248
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, repl
Affects Versions: 3.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.1.0


This is the case where the events were deleted on source because of old event 
purging and hence min(source event id) > target event id (last replicated event 
id).

Repl dump should fail in this case so that user can drop the database and 
bootstrap again.

Cleaner thread is concurrently removing the expired events from 
NOTIFICATION_LOG table. So, it is necessary to check if the current dump missed 
any event while dumping. After fetching events in batches, we shall check if it 
is fetched in contiguous sequence of event id. If it is not in contiguous 
sequence, then likely some events missed in the dump and hence throw error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19249) Replication: The WITH clause is not passing the configuration to Task correctly in all cases

2018-04-19 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-19249:
---

 Summary: Replication: The WITH clause is not passing the 
configuration to Task correctly in all cases
 Key: HIVE-19249
 URL: https://issues.apache.org/jira/browse/HIVE-19249
 Project: Hive
  Issue Type: Bug
  Components: repl
Affects Versions: 3.0.0, 3.1.0
Reporter: Vaibhav Gumashta


When running repl load like following:
{code}
REPL LOAD `repldb_kms207` FROM 
'hdfs://url:8020/apps/hive/repl/f8b057a7-c3f2-43bd-8baa-f7408a9008fc' WITH 
('hive.exec.parallel'='true','hive.distcp.privileged.doAs'='beacon','hive.metastore.uris'='thrift://metastore-url:9083','hive.metastore.warehouse.dir'='s3a://s3-warehouse','hive.warehouse.subdir.inherit.perms'='false','hive.repl.replica.functions.root.dir'='s3a://s3-warehouse','fs.s3a.bucket.ss-datasets.endpoint'='s3-bucket-endpoint','fs.s3a.impl.disable.cache'='true','fs.s3a.server-side-encryption-algorithm'='SSE-KMS','fs.s3a.server-side-encryption.key'='encr-key','distcp.options.pp'='','distcp.options.pg'='','distcp.options.pu'='');
{code}

the task that get created need to use the configs that are passed in the USING 
clause. However, in some cases the wrong config object gets used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19250) Schema column definitions inconsistencies in MySQL

2018-04-19 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-19250:


 Summary: Schema column definitions inconsistencies in MySQL
 Key: HIVE-19250
 URL: https://issues.apache.org/jira/browse/HIVE-19250
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


There are some inconsistencies in column definitions in MySQL between a schema 
that was upgraded to 2.1 (from an older release) vs installing the 2.1.0 schema 
directly.
>   `CQ_TBLPROPERTIES` varchar(2048) DEFAULT NULL,
117d117
<   `CQ_TBLPROPERTIES` varchar(2048) DEFAULT NULL,
135a136
>   `CC_TBLPROPERTIES` varchar(2048) DEFAULT NULL,
143d143
<   `CC_TBLPROPERTIES` varchar(2048) DEFAULT NULL,
156c156
<   `CTC_TXNID` bigint(20) DEFAULT NULL,
---
>   `CTC_TXNID` bigint(20) NOT NULL,
158c158
<   `CTC_TABLE` varchar(256) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT 
NULL,
---
>   `CTC_TABLE` varchar(256) DEFAULT NULL,
476c476
<   `TBL_NAME` varchar(256) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT 
NULL,
---
>   `TBL_NAME` varchar(256) DEFAULT NULL,
664c664
<   KEY `PCS_STATS_IDX` (`DB_NAME`,`TABLE_NAME`,`COLUMN_NAME`,`PARTITION_NAME`),
---
>   KEY `PCS_STATS_IDX` (`DB_NAME`,`TABLE_NAME`,`COLUMN_NAME`,`PARTITION_NAME`) 
> USING BTREE,
768c768
<   `PARAM_VALUE` mediumtext,
---
>   `PARAM_VALUE` mediumtext CHARACTER SET latin1 COLLATE latin1_bin,
814c814
<   `PARAM_VALUE` mediumtext,
---
>   `PARAM_VALUE` mediumtext CHARACTER SET latin1 COLLATE latin1_bin,
934c934
<   `PARAM_VALUE` mediumtext,
---
>   `PARAM_VALUE` mediumtext CHARACTER SET latin1 COLLATE latin1_bin,
1066d1065
<   `TXN_HEARTBEAT_COUNT` int(11) DEFAULT NULL,
1067a1067
>   `TXN_HEARTBEAT_COUNT` int(11) DEFAULT NULL,
1080c1080
<   `TC_TXNID` bigint(20) DEFAULT NULL,
---
>   `TC_TXNID` bigint(20) NOT NULL,
1082c1082
<   `TC_TABLE` varchar(128) DEFAULT NULL,
---
>   `TC_TABLE` varchar(128) NOT NULL,
1084c1084
<   `TC_OPERATION_TYPE` char(1) DEFAULT NULL,
---
>   `TC_OPERATION_TYPE` char(1) NOT NULL,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #333: HIVE-19197: TestReplicationScenarios is flaky

2018-04-19 Thread sankarh 
Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/333


---

Builds failing in ptest

2018-04-19 Thread Deepak Jaiswal 
Hi All,

The builds are failing on Jenkins with this exception.

Checking PreCommit-HIVE-Build queue...
Exception in thread "main" java.lang.IllegalStateException: 502 Proxy Error
at 
org.apache.hive.ptest.api.client.JenkinsQueueUtil.httpGet(JenkinsQueueUtil.java:130)
at 
org.apache.hive.ptest.api.client.JenkinsQueueUtil.isJiraAlreadyInQueue(JenkinsQueueUtil.java:72)
at 
org.apache.hive.ptest.api.client.PTestClient.main(PTestClient.java:325)


It originates from a recent change to handle ptest queue in HIVE-19077.
Currently, all the tests enqueued are just failing one by one.

Regards,
Deepak

Re: Review Request 66416: HIVE-17647 DDLTask.generateAddMmTasks(Table tbl) and other random code should not start transactions

2018-04-19 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66416/#review201543
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 1436 (patched)


getTableWriteId() ensures you only allocate 1 write id per txn


- Eugene Koifman


On April 18, 2018, 4:16 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66416/
> ---
> 
> (Updated April 18, 2018, 4:16 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties d26f0ccb17 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java f53afaff2b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ImportCommitTask.java b3c62ad1a8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ImportCommitWork.java a119250464 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 10a2ed2663 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 4760b85da0 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 009a890888 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 85d1cff320 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 49a3464776 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> ac44be5e0b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
> bdecbaf144 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java d7b224772d 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DDLDesc.java 65f4cf233b 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 0926663eab 
>   ql/src/test/results/clientpositive/mm_conversions.q.out 4754710291 
> 
> 
> Diff: https://reviews.apache.org/r/66416/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date/Timestamp

2018-04-19 Thread Gopal V (JIRA) 
Gopal V created HIVE-19247:
--

 Summary: StatsOptimizer: Missing stats fast-path for Date/Timestamp
 Key: HIVE-19247
 URL: https://issues.apache.org/jira/browse/HIVE-19247
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Affects Versions: 2.3.2, 2.2.0, 3.0.0
Reporter: Gopal V


{code}
2018-04-19T18:57:24,268 DEBUG [67259108-c184-4c92-9e18-9e296922 
HiveServer2-Handler-Pool: Thread-73]: optimizer.StatsOptimizer 
(StatsOptimizer.java:process(614)) - Unsupported type: date encountered in 
metadata optimizer for column : jour
{code}

{code}
if (udaf instanceof GenericUDAFMin) {
ExprNodeColumnDesc colDesc = 
(ExprNodeColumnDesc)exprMap.get(((ExprNodeColumnDesc)aggr.getParameters().get(0)).getColumn());
String colName = colDesc.getColumn();
StatType type = getType(colDesc.getTypeString());
if (!tbl.isPartitioned()) {
  if (!StatsSetupConst.areColumnStatsUptoDate(tbl.getParameters(), 
colName)) {
Logger.debug("Stats for table : " + tbl.getTableName() + " 
column " + colName
+ " are not up to date.");
return null;
  }
  ColumnStatisticsData statData = 
hive.getMSC().getTableColumnStatistics(
  tbl.getDbName(), tbl.getTableName(), 
Lists.newArrayList(colName))
  .get(0).getStatsData();
  String name = colDesc.getTypeString().toUpperCase();
  switch (type) {
case Integeral: {
  LongSubType subType = LongSubType.valueOf(name);
  LongColumnStatsData lstats = statData.getLongStats();
  if (lstats.isSetLowValue()) {
oneRow.add(subType.cast(lstats.getLowValue()));
  } else {
oneRow.add(null);
  }
  break;
}
case Double: {
  DoubleSubType subType = DoubleSubType.valueOf(name);
  DoubleColumnStatsData dstats = statData.getDoubleStats();
  if (dstats.isSetLowValue()) {
oneRow.add(subType.cast(dstats.getLowValue()));
  } else {
oneRow.add(null);
  }
  break;
}
default: // unsupported type
  Logger.debug("Unsupported type: " + colDesc.getTypeString() + 
" encountered in " +
  "metadata optimizer for column : " + colName);
  return null;
  }
}
{code}

{code}
enum StatType{
  Integeral,
  Double,
  String,
  Boolean,
  Binary,
  Unsupported
}

enum LongSubType {
  BIGINT { @Override
  Object cast(long longValue) { return longValue; } },
  INT { @Override
  Object cast(long longValue) { return (int)longValue; } },
  SMALLINT { @Override
  Object cast(long longValue) { return (short)longValue; } },
  TINYINT { @Override
  Object cast(long longValue) { return (byte)longValue; } };

  abstract Object cast(long longValue);
}
{code}

Date/Timestamp are stored as Integral stats (& also the typo there).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66720: HIVE-17657 export/import for MM tables is broken

2018-04-19 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66720/
---

Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java ce683c8a8d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java aba65918f8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 5fbe045df5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java ce0757cba2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
d3c62a2775 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
ac44be5e0b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
820046388a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/PartitionExport.java 
5844f3d97f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java 
abb2e8874b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java 
866d3513b1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CopyWork.java c0e4a43d9c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExportWork.java 72ce79836c 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 12d57c6feb 
  ql/src/test/queries/clientpositive/mm_exim.q c47342bd23 
  ql/src/test/results/clientpositive/llap/mm_exim.q.out 1f40754373 


Diff: https://reviews.apache.org/r/66720/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently

2018-04-19 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-19252:
---

 Summary: TestJdbcWithMiniKdcCookie.testCookieNegative is failing 
consistently
 Key: HIVE-19252
 URL: https://issues.apache.org/jira/browse/HIVE-19252
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Ashutosh Chauhan
Assignee: Thejas M Nair


For last 8 builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-19 Thread Eugene Koifman 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201574
---




ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 1324 (patched)


why not use TxnUtils.createValidCompactWriteIdList() and 
ValidCompactorWriteIdList.  It sets HWM to minOpen-1 - that is what it's for



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1248 (patched)


should 'newBaseDir'?



ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
Lines 1250 (patched)


Why is this OK?  The query produced no output?


- Eugene Koifman


On April 17, 2018, 6:46 p.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 17, 2018, 6:46 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Created] (HIVE-19251) ObjectStore.getNextNotification with LIMIT should use less memory

2018-04-19 Thread Daniel Dai (JIRA) 
Daniel Dai created HIVE-19251:
-

 Summary: ObjectStore.getNextNotification with LIMIT should use 
less memory
 Key: HIVE-19251
 URL: https://issues.apache.org/jira/browse/HIVE-19251
 Project: Hive
  Issue Type: Bug
  Components: repl, Standalone Metastore
Reporter: Daniel Dai
Assignee: Daniel Dai


Experience OOM when Hive metastore try to retrieve huge amount of notification 
logs even there's limit clause. Hive shall only retrieve necessary rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66663: HIVE-19171 Persist runtime statistics in metastore

2018-04-19 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3/#review201594
---




ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java
Line 126 (original), 122 (patched)


this should pass in default values from conf.



ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java
Line 96 (original), 97 (patched)


Cache size is used as number of entries. I see your intention but I am not 
sure it will work as intended. Also, its easier to reason about cache size as 
number of entries. So, lets use that.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Line 11643 (original), 11627 (patched)


Deletion shouldn't be based on total weight. It may skew entries from one 
large entry.
It could be based on min weight, that is delete all entries whose weight < 
minWeight.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
Line 1636 (original), 1636 (patched)


This method can be renamed : deleteRuntimeStats() ?



standalone-metastore/src/main/thrift/hive_metastore.thrift
Line 2180 (original), 2180 (patched)


should throw MetaException.



standalone-metastore/src/main/thrift/hive_metastore.thrift
Line 2181 (original), 2181 (patched)


create a param request object for  minCreateTime,  i32 maxCount to follow 
req/resp object convention.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java
Lines 59 (patched)


delete call can return how many rows it deleted. Useful for logging.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java
Lines 63 (patched)


update message.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RuntimeStatsCleanerTask.java
Lines 66 (patched)


update message.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 584 (patched)


this wil turn off cleaner thread. Lets use default value of 86400



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
Lines 586 (patched)


sum of weights is little hard to reason about. Lets just have max weight as 
config.


- Ashutosh Chauhan


On April 19, 2018, 3:54 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3/
> ---
> 
> (Updated April 19, 2018, 3:54 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-19171
> https://issues.apache.org/jira/browse/HIVE-19171
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * wip patch
> * also contains HIVE-19009
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 73492ff99c 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  801de7aca2 
>   itests/src/test/resources/testconfiguration.properties d26f0ccb17 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> fad8c0ffe9 
>   metastore/scripts/upgrade/derby/056-HIVE-19171.derby.sql PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 9ca8b0007e 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java f761fffbf1 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> e15c5b7b66 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java 
> e87bbceb7a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java
>  c3dc848a32 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsMap.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/RuntimeStatsPersister.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java e15a49f838 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java 
> a61a47e390 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 5b7f4c3560 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 54b705db6e 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-19 Thread Sergey Shelukhin 


> On April 19, 2018, 10:48 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 1324 (patched)
> > 
> >
> > why not use TxnUtils.createValidCompactWriteIdList() and 
> > ValidCompactorWriteIdList.  It sets HWM to minOpen-1 - that is what it's for

that seems to require different types that this code doesn't have... as far as 
I see it would need to call metastore or rebuild another object from scratch. 
There are too many (Table|Valid|WriteIds|*){3,} classes :)


> On April 19, 2018, 10:48 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java
> > Lines 1250 (patched)
> > 
> >
> > Why is this OK?  The query produced no output?

This is the same commitJob in the compactor. I'm not sure if it can happen... 
perhaps from a number of empty inserts/ctas?


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/#review201574
---


On April 18, 2018, 1:46 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66485/
> ---
> 
> (Updated April 18, 2018, 1:46 a.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b8d948d3d3 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  82ba775286 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4acdd9b49b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b1c2288d01 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 22765b8e63 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
>   
> standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  cb1d40a4a8 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  7b02865e18 
>   
> storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java
>  107ea9028a 
> 
> 
> Diff: https://reviews.apache.org/r/66485/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 66485: HIVE-19124 implement a basic major compactor for MM tables

2018-04-19 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66485/
---

(Updated April 20, 2018, 1:42 a.m.)


Review request for hive and Eugene Koifman.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 536c7b427f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 82ba775286 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 9cb2ff1015 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c8cb8a40b4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b1c2288d01 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 22765b8e63 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java fe0aaa4ff5 
  
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 cb1d40a4a8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
 7b02865e18 
  
storage-api/src/java/org/apache/hadoop/hive/common/ValidReaderWriteIdList.java 
107ea9028a 


Diff: https://reviews.apache.org/r/66485/diff/6/

Changes: https://reviews.apache.org/r/66485/diff/5-6/


Testing
---


Thanks,

Sergey Shelukhin