[jira] [Created] (HIVE-19187) Update Druid Storage Handler to Druid 0.12.0

2018-04-11 Thread slim bouguerra (JIRA) 
slim bouguerra created HIVE-19187:
-

 Summary: Update Druid Storage Handler to Druid 0.12.0
 Key: HIVE-19187
 URL: https://issues.apache.org/jira/browse/HIVE-19187
 Project: Hive
  Issue Type: Bug
  Components: Druid integration
Reporter: slim bouguerra
Assignee: slim bouguerra
 Fix For: 3.1.0


Current used Druid Version is 0.11.0
This Patch updates the Druid version to the most recent version 0.12.0




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19186) Multi Table INSERT statements query have a flaw for partitioned table when INSERT INTO and INSERT OVERWRITE are used

2018-04-11 Thread Steve Yeom (JIRA) 
Steve Yeom created HIVE-19186:
-

 Summary: Multi Table INSERT statements query have a flaw for 
partitioned table when INSERT INTO and INSERT OVERWRITE are used
 Key: HIVE-19186
 URL: https://issues.apache.org/jira/browse/HIVE-19186
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 3.0.0
Reporter: Steve Yeom
 Fix For: 3.0.0


One problem test case is: 

create table intermediate(key int) partitioned by (p int) stored as orc;
insert into table intermediate partition(p='455') select distinct key from src 
where key >= 0 order by key desc limit 2;
insert into table intermediate partition(p='456') select distinct key from src 
where key is not null order by key asc limit 2;
insert into table intermediate partition(p='457') select distinct key from src 
where key >= 100 order by key asc limit 2;

from intermediate
insert into table multi_partitioned partition(p=2) select p, key
insert overwrite table multi_partitioned partition(p=1) select key, p;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19185) Orc files can support zstd or lz4 ？

2018-04-11 Thread gehaijiang (JIRA) 
gehaijiang created HIVE-19185:
-

 Summary: Orc files can support zstd or lz4 ？
 Key: HIVE-19185
 URL: https://issues.apache.org/jira/browse/HIVE-19185
 Project: Hive
  Issue Type: Wish
Reporter: gehaijiang


orc file  ： high level compression (one of NONE, ZLIB, SNAPPY)  ，  When can we 
support LZ4  or   ZSTD？



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19184) Hive 3.0.0 release preparation

2018-04-11 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-19184:
--

 Summary: Hive 3.0.0 release preparation
 Key: HIVE-19184
 URL: https://issues.apache.org/jira/browse/HIVE-19184
 Project: Hive
  Issue Type: Task
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0


Need to do bunch of things to prepare branch-3 for release e.g.
* Update pom.xml to delete SNAPSHOT
* Update .reviewboardrc
* Remove storage-api module to build
* Change storage-api depdency etc



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

2018-04-11 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
---

(Updated April 11, 2018, 10:53 p.m.)


Review request for hive and Thejas Nair.


Changes
---

Changed some logging to trace level


Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-19126


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 6f4ec6f1ea 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 2f7fa24558 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 0bbaf7e459 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856de87 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 89b400697b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f007261daf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66503/diff/5/

Changes: https://reviews.apache.org/r/66503/diff/4-5/


Testing
---


Thanks,

Vaibhav Gumashta

Re: Review Request 66503: HIVE-19126: CachedStore: Use memory estimation to limit cache size during prewarm

2018-04-11 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66503/
---

(Updated April 11, 2018, 10:38 p.m.)


Review request for hive and Thejas Nair.


Bugs: HIVE-19126
https://issues.apache.org/jira/browse/HIVE-19126


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-19126


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
 6f4ec6f1ea 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 2f7fa24558 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestIncrementalObjectSizeEstimator.java
 0bbaf7e459 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856de87 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 89b400697b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 f007261daf 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/SizeValidator.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/66503/diff/4/

Changes: https://reviews.apache.org/r/66503/diff/3-4/


Testing
---


Thanks,

Vaibhav Gumashta

Review Request 66571: HIVE-19161: Add authorizations to information schema

2018-04-11 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66571/
---

Review request for hive.


Repository: hive-git


Description
---

See HIVE-19161


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0627c35 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 801de7a 
  itests/hive-unit/pom.xml f473d25 
  
itests/hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java
 PRE-CREATION 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql 339 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1f 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveResourceACLs.java
 53e221f 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 6308c5c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 450da4f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 ebbf465 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 b2c40c2 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 2056930 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 f6c46ee 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 c47856d 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 7450439 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 304f567 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 85c6727 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7d37262 


Diff: https://reviews.apache.org/r/66571/diff/1/


Testing
---


Thanks,

Daniel Dai

May 2018 Hive User Group Meeting

2018-04-11 Thread Sahil Takiar 
Hi all,

I'm happy to announce that the Hive community is organizing a Hive user
group meeting in the Bay Area next month. The details can be found at
https://www.meetup.com/Hive-User-Group-Meeting/events/249641278/

The format of this meetup will be slightly different from previous ones.
There will be one hour dedicated to lightning talks, followed by a group
discussion on the future of the Hive Metastore.

We are inviting talk proposals from Hive users as well as developers at
this time. Please contact either myself (takiar.sa...@gmail.com), Vihang
Karajgaonkar (vih...@cloudera.com), or Peter Vary (pv...@cloudera.com) with
proposals. We currently have 5 openings.

Please let me know if you have any questions or suggestions.

Thanks,
Sahil

[jira] [Created] (HIVE-19183) Confluence Documentation for LDAP Config has unfortunate bug

2018-04-11 Thread Ben M (JIRA) 
Ben M created HIVE-19183:


 Summary: Confluence Documentation for LDAP Config has unfortunate 
bug
 Key: HIVE-19183
 URL: https://issues.apache.org/jira/browse/HIVE-19183
 Project: Hive
  Issue Type: Bug
  Components: Documentation
Reporter: Ben M


Not sure how else to notify of this, apparently can't add comments (SAD!) or 
edit confluence.

 

Please see: 
[https://cwiki.apache.org/confluence/display/Hive/User+and+Group+Filter+Support+with+LDAP+Atn+Provider+in+HiveServer2]

Note that configuration property defined as: 
{{hive.server.authentication.ldap.groupClassKey}} SHOULD be

{{hive.server.authentication.ldap.groupClassKey}} (note {{server}} vs. 
{{server2}})

sadly spent a few hours today trying to suss out why my config wasn't working 
until i realized that my overcaution and copying the exact key from the docs 
was the root cause :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19182) Code clean up of HiveMetaStoreChecker.java

2018-04-11 Thread Janaki Lahorani (JIRA) 
Janaki Lahorani created HIVE-19182:
--

 Summary: Code clean up of HiveMetaStoreChecker.java
 Key: HIVE-19182
 URL: https://issues.apache.org/jira/browse/HIVE-19182
 Project: Hive
  Issue Type: Improvement
Reporter: Janaki Lahorani
Assignee: Janaki Lahorani


HiveMetaStoreChecker.java has code like:
result.getTablesNotOnFs().add(table.getTableName());
result.getPartitionsNotOnFs().add(pr);
result.getPartitionsNotInMs().add(pr);

It is not a good practice to change internal states like this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19181) Remove BreakableService (unused class)

2018-04-11 Thread Thejas M Nair (JIRA) 
Thejas M Nair created HIVE-19181:


 Summary: Remove BreakableService (unused class)
 Key: HIVE-19181
 URL: https://issues.apache.org/jira/browse/HIVE-19181
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2, 3.0.0
Reporter: Thejas M Nair


BreakableService.java is not used anywhere




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 66567: Migrate to Murmur hash for shuffle and bucketing

2018-04-11 Thread Deepak Jaiswal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66567/
---

Review request for hive, Eugene Koifman and Jason Dere.


Bugs: HIVE-18910
https://issues.apache.org/jira/browse/HIVE-18910


Repository: hive-git


Description
---

Hive uses JAVA hash which is not as good as murmur for better distribution and 
efficiency in bucketing a table.
Migrate to murmur hash but still keep backward compatibility for existing users 
so that they dont have to reload the existing tables.

To keep backward compatibility, bucket_version is added as a table property, 
resulting in high number of result updates.


Diffs
-

  hbase-handler/src/test/results/positive/external_table_ppd.q.out cdc43ee560 
  hbase-handler/src/test/results/positive/hbase_binary_storage_queries.q.out 
153613e6d0 
  hbase-handler/src/test/results/positive/hbase_ddl.q.out ef3f5f704e 
  hbase-handler/src/test/results/positive/hbasestats.q.out 5d000d2f4f 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
 924e233293 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolver.java
 5dd0b8ea5b 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/BucketIdResolverImpl.java
 7c2cadefa7 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/worker/MutatorCoordinator.java
 ad14c7265f 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 3733e3d02f 
  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/mutate/worker/TestBucketIdResolverImpl.java
 03c28a33c8 
  
hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java
 996329195c 
  
hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
 f9ee9d9a03 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
 caa00292b8 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
ab8ad77074 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
 2b28a6677e 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
 cdb67dd786 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
 2c23a7e94f 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 a1be085ea5 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
353b890b7c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 5966740f88 
  itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c084fa054c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d59bf1fb6e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java d4363fdf91 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/keyseries/VectorKeySeriesSerializedImpl.java
 86f466fc4e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 4077552a56 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
 1bc3fdabac 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java a51fdd322f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 
7121bceb22 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
 5f65f638ca 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/PrunerOperatorFactory.java 
2be3c9b9a2 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
 1c5656267d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionTimeGranularityOptimizer.java
 0e995d79d2 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/AnnotateWithOpTraits.java
 3c8e61d47b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java
 69d9f3125a 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
d3fbf07de4 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
 7b1fd5f206 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 10982ddbd1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/OpTraits.java 9621c3be53 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java dde20ed56e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java aa3c72bc6d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 25b91899de 
  ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java 
adea3b53a9 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java 
7cd571815d

[jira] [Created] (HIVE-19180) beeline: user interrupt should halt execution

2018-04-11 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-19180:
---

 Summary: beeline: user interrupt should halt execution 
 Key: HIVE-19180
 URL: https://issues.apache.org/jira/browse/HIVE-19180
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Zoltan Haindrich


I'm running a queryfile with {{--force}} enabled.

By interrupting the program (CTRL+C) I can only abort the execution of the 
actual query; but the queryFile execution is continued regardless that I wanted 
to abort.

for example, for the following query file; I'm able to interrupt all inserts; 
but even the select is executed...and I think the exit code should be non-zero 
as well.

{code}
drop table if exists ll1;
create table ll1( a integer);
insert into ll1 values (1);
insert into ll1 values (2);
insert into ll1 values (3);
select * from ll1;
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19179) Organize Spark metrics into multiple groups

2018-04-11 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-19179:
---

 Summary: Organize Spark metrics into multiple groups
 Key: HIVE-19179
 URL: https://issues.apache.org/jira/browse/HIVE-19179
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


Right now all Spark metrics are under a single group named {{SPARK}}, it would 
be good to divide them into multiple groups in order to differentiate between 
input metrics, shuffle read metrics, shuffle write metrics, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19177) ObjectStore.setConf() is doing dangerous work while holding global lock

2018-04-11 Thread Alexander Kolbasov (JIRA) 
Alexander Kolbasov created HIVE-19177:
-

 Summary: ObjectStore.setConf() is doing dangerous work while 
holding global lock
 Key: HIVE-19177
 URL: https://issues.apache.org/jira/browse/HIVE-19177
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 2.0.2, 3.0.0, 3.1.0
Reporter: Alexander Kolbasov


The {{ObjectStore.setConf()}} function grabs static {{pmfPropLock}} and then 
calls {{initialize}} which goes through DataNucleus, accesses database, waits 
on DB thread pools, retries with sleep, etc, all while holding the lock.

This is rather dangerous and expensive since no one else can call setConf at 
the same time on a different instance. All of these should be done without 
holding the lock.

[~vihangk1] FYI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19178) TestMiniTezCliDriver.testCliDriver[explainanalyze_5] failure

2018-04-11 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-19178:
--

 Summary: TestMiniTezCliDriver.testCliDriver[explainanalyze_5] 
failure
 Key: HIVE-19178
 URL: https://issues.apache.org/jira/browse/HIVE-19178
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0


I have verified that this failure is due to HIVE-18825.

Error stack:
{code}
java.lang.IllegalStateException: calling recordValidTxn() more than once in the 
same txnid:5
at org.apache.hadoop.hive.ql.Driver.acquireLocks(Driver.java:1439)
at org.apache.hadoop.hive.ql.Driver.lockAndRespond(Driver.java:1624)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1794)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1527)
at 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:137)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:287)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:635)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1655)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1602)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1597)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:200)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
at 
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
at 
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
at 
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at 
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver(TestMiniTezCliDriver.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
at

Re: Review Request 66533: HIVE-19154 Poll notification events to invalidate the results cache

2018-04-11 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66533/#review200927
---


Ship it!




Ship It!

- Gopal V


On April 11, 2018, 6:01 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66533/
> ---
> 
> (Updated April 11, 2018, 6:01 p.m.)
> 
> 
> Review request for hive, Gopal V and Thejas Nair.
> 
> 
> Bugs: HIVE-19154
> https://issues.apache.org/jira/browse/HIVE-19154
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Create NotificationEventPoll to periodically query for notification events, 
> and pass the events to any registered EventConsumers.
> - Create InvalidationEventConsumer in QueryResultsCache to use the events to 
> invalidate any results cache entries using the updated table.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e540d023bd 
>   itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 3cdad284ef 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
> b1a3646624 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/events/EventConsumer.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/events/NotificationEventPoll.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 10982ddbd1 
>   ql/src/test/queries/clientpositive/results_cache_invalidation2.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/results_cache_invalidation2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/results_cache_invalidation2.q.out 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 47f84b5e73 
> 
> 
> Diff: https://reviews.apache.org/r/66533/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

Re: Review Request 66514: HIVE-17645 MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)

2018-04-11 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66514/
---

(Updated April 11, 2018, 7:58 p.m.)


Review request for hive, Eugene Koifman and Sergey Shelukhin.


Changes
---

Added comment to SessionState.getTxnMgr() about avoiding use of this call.


Repository: hive-git


Description
---

Replace usage of SessionState.getTxnMgr() from several places, by doing some 
refactoring to make the TxnManager available in fields passed in during 
construction/initialization:
- SemanticAnalyzer.genFileSinkPlan()
- ReplicationSemanticAnalyzer.analyzeReplLoad()
- LoadSemanticAnalyzer.analyzeExternal()
- ImportSemanticAnalyzer.prepareImport()
- DDLSemanticAnalyzer.handleTransactionalTable()


Diffs (updated)
-

  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
3aec46be51 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java bda2af3a04 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java a8d851fd81 
  ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/ReplLoadTask.java 
6b333d7184 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadConstraint.java
 60c85f58e5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/LoadFunction.java
 bc7d0ad0b9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadPartitions.java
 06adc64727 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/table/LoadTable.java
 1395027159 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/bootstrap/load/util/Context.java
 bb51f36a25 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7a7bdea89d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
f38b0bc546 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
8b639f7922 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
e49089b91e 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java
 e5af95b121 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java 
79b2e48ee2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 10982ddbd1 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/MessageHandler.java
 3ccd639d62 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/message/TableHandler.java 
4cd75d8128 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6003ced27e 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
fe570f0f8e 


Diff: https://reviews.apache.org/r/66514/diff/3/

Changes: https://reviews.apache.org/r/66514/diff/2-3/


Testing
---


Thanks,

Jason Dere

[jira] [Created] (HIVE-19176) Add HoS support to progress bar on Beeline client

2018-04-11 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-19176:
---

 Summary: Add HoS support to progress bar on Beeline client
 Key: HIVE-19176
 URL: https://issues.apache.org/jira/browse/HIVE-19176
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


Make whats was done in HIVE-15473 work for HoS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19175) TestMiniLlapLocalCliDriver.testCliDriver update_access_time_non_current_db failing

2018-04-11 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-19175:
--

 Summary: TestMiniLlapLocalCliDriver.testCliDriver 
update_access_time_non_current_db failing
 Key: HIVE-19175
 URL: https://issues.apache.org/jira/browse/HIVE-19175
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Vineet Garg
Assignee: Vineet Garg
 Fix For: 3.0.0


Caused by HIVE-18060. Instead of generating golden file under 
clientpositive/llap it is under clientpositive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66533: HIVE-19154 Poll notification events to invalidate the results cache

2018-04-11 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66533/
---

(Updated April 11, 2018, 6:01 p.m.)


Review request for hive, Gopal V and Thejas Nair.


Changes
---

Using SessionState query timestamp as the cache entry's query time.


Bugs: HIVE-19154
https://issues.apache.org/jira/browse/HIVE-19154


Repository: hive-git


Description
---

- Create NotificationEventPoll to periodically query for notification events, 
and pass the events to any registered EventConsumers.
- Create InvalidationEventConsumer in QueryResultsCache to use the events to 
invalidate any results cache entries using the updated table.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e540d023bd 
  itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 3cdad284ef 
  ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
b1a3646624 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/events/EventConsumer.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/events/NotificationEventPoll.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 10982ddbd1 
  ql/src/test/queries/clientpositive/results_cache_invalidation2.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/results_cache_invalidation2.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/results_cache_invalidation2.q.out 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 47f84b5e73 


Diff: https://reviews.apache.org/r/66533/diff/2/

Changes: https://reviews.apache.org/r/66533/diff/1-2/


Testing
---


Thanks,

Jason Dere

Re: Review Request 66533: HIVE-19154 Poll notification events to invalidate the results cache

2018-04-11 Thread Jason Dere 


> On April 11, 2018, 5:26 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java
> > Lines 470 (patched)
> > 
> >
> > SessionState.get().getQueryCurrentTimestamp()
> > 
> > Possibly pass it in via QueryInfo?

Good suggestion, will make the change.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66533/#review200887
---


On April 10, 2018, 7:19 p.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66533/
> ---
> 
> (Updated April 10, 2018, 7:19 p.m.)
> 
> 
> Review request for hive, Gopal V and Thejas Nair.
> 
> 
> Bugs: HIVE-19154
> https://issues.apache.org/jira/browse/HIVE-19154
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> - Create NotificationEventPoll to periodically query for notification events, 
> and pass the events to any registered EventConsumers.
> - Create InvalidationEventConsumer in QueryResultsCache to use the events to 
> invalidate any results cache entries using the updated table.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e540d023bd 
>   itests/src/test/resources/testconfiguration.properties 48d62a8bf9 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 3cdad284ef 
>   ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java 
> b1a3646624 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/events/EventConsumer.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/events/NotificationEventPoll.java
>  PRE-CREATION 
>   ql/src/test/queries/clientpositive/results_cache_invalidation2.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/results_cache_invalidation2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/results_cache_invalidation2.q.out 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 47f84b5e73 
> 
> 
> Diff: https://reviews.apache.org/r/66533/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jason Dere
> 
>

[jira] [Created] (HIVE-19174) Enhance HMS to support drop of partitions with partition names as input

2018-04-11 Thread Janaki Lahorani (JIRA) 
Janaki Lahorani created HIVE-19174:
--

 Summary: Enhance HMS to support drop of partitions with partition 
names as input
 Key: HIVE-19174
 URL: https://issues.apache.org/jira/browse/HIVE-19174
 Project: Hive
  Issue Type: Improvement
Reporter: Janaki Lahorani
Assignee: Janaki Lahorani






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66557: PidFilePatternConverter not rolling log according to their pid

2018-04-11 Thread Bohdan Chupika 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66557/
---

(Updated Апрель 11, 2018, 4 п.п.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17477
https://issues.apache.org/jira/browse/HIVE-17477


Repository: hive-git


Description (updated)
---

PROBLEM:
We can use pid in th filePattern:

appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d
{-MM-dd}

.%pid

But when its time roll the logs at the end of the RollingAppender just renames 
hive.log by applying the pattern described above and keeps logs that are not 
related to the process this log was named by. The issue is that all processes 
are writing to the same log and we cannot separate them by pid.

SOLUTION:
I have discovered apache log4j2 project and documentation and found that 
existed PidFilePatternConverter allowed appending PID only when 
TimeBasedTriggeringPolicy was occured. I have  ported 
https://github.com/apache/logging-log4j2/blob/master/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/ProcessIdPatternConverter.java
 and configured new RoutingAppender in hive-log4j2.properties.

I have removed old PidFilePatternConverter.java file and create new 
ProcessIdPatternConverter.java because first class contain in name 
"FilePattern" and I think it could be confused. PidFilePatternConverter's 
implementation required adding %pid to appender.XXX.filePattern in 
hive-log4j2.properties. But ProcessIdPatternConverter depends not on 
filePattern, it depends on LogEvents, and %pid should be added to 
rolling.layout.pattern field.


Diffs
-

  common/src/main/resources/hive-log4j2.properties 
b1719907571ead14686e7fcc14a0687158dec941 
  llap-server/src/main/resources/llap-cli-log4j2.properties 
687c97399a6632205318c24665ed9a97dc30c552 
  ql/src/java/org/apache/hadoop/hive/ql/log/PidFilePatternConverter.java 
c49f53fd3cdcf71ebca36dec185021e18d0b1ef9 
  ql/src/java/org/apache/hadoop/hive/ql/log/ProcessIdPatternConverter.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/ProcessIdPatternConverterTest.java 
PRE-CREATION 
  standalone-metastore/src/main/resources/metastore-log4j2.properties 
ec5039b4fbdf880fee2a3f86ec3402ba02f3d21d 


Diff: https://reviews.apache.org/r/66557/diff/1/


Testing
---


Thanks,

Bohdan Chupika

Review Request 66557: PidFilePatternConverter not rolling log according to their pid

2018-04-11 Thread Bohdan Chupika 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66557/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17477
https://issues.apache.org/jira/browse/HIVE-17477


Repository: hive-git


Description
---

PROBLEM:
We can use pid in th filePattern:

appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d
{-MM-dd}

.%pid

But when its time roll the logs at the end of the RollingAppender just renames 
hive.log by applying the pattern described above and keeps logs that are not 
related to the process this log was named by. The issue is that all processes 
are writing to the same log and we cannot separate them by pid.

SOLUTION:
I have discovered apache log4j2 project and documentation and found that 
existed PidFilePatternConverter allowed appending PID only when 
TimeBasedTriggeringPolicy was occured. I have created ported 
https://github.com/apache/logging-log4j2/blob/master/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/ProcessIdPatternConverter.java
 and configured new RoutingAppender in hive-log4j2.properties.

I have removed old PidFilePatternConverter.java file and create new 
ProcessIdPatternConverter.java because first class contain in name 
"FilePattern" and I think it could be confused. PidFilePatternConverter's 
implementation required adding %pid to appender.XXX.filePattern in 
hive-log4j2.properties. But ProcessIdPatternConverter depends not on 
filePattern, it depends on LogEvents, and %pid should be added to 
rolling.layout.pattern field.


Diffs
-

  common/src/main/resources/hive-log4j2.properties 
b1719907571ead14686e7fcc14a0687158dec941 
  llap-server/src/main/resources/llap-cli-log4j2.properties 
687c97399a6632205318c24665ed9a97dc30c552 
  ql/src/java/org/apache/hadoop/hive/ql/log/PidFilePatternConverter.java 
c49f53fd3cdcf71ebca36dec185021e18d0b1ef9 
  ql/src/java/org/apache/hadoop/hive/ql/log/ProcessIdPatternConverter.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/log/ProcessIdPatternConverterTest.java 
PRE-CREATION 
  standalone-metastore/src/main/resources/metastore-log4j2.properties 
ec5039b4fbdf880fee2a3f86ec3402ba02f3d21d 


Diff: https://reviews.apache.org/r/66557/diff/1/


Testing
---


Thanks,

Bohdan Chupika

[jira] [Created] (HIVE-19173) Add Storage Handler runtime information as part of DESCRIBE EXTENDED

2018-04-11 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-19173:
---

 Summary: Add Storage Handler runtime information as part of 
DESCRIBE EXTENDED
 Key: HIVE-19173
 URL: https://issues.apache.org/jira/browse/HIVE-19173
 Project: Hive
  Issue Type: Task
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


Follow up for https://issues.apache.org/jira/browse/HIVE-18976 
Kafka Indexing Service in Druid has a runtime state associated with it. 
Druid publishes this runtime state as KafkaSupervisorReport which has latest 
offsets as reported by Kafka, the consumer lag per partition, as well as the 
aggregate lag of all partitions.

This information is quite useful to know whether a kafka-indexing-service 
backed table has latest info or not. 

This task is to add a this information as part of the output of DESCRIBE 
EXTENDED statement



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19172) NPE due to null EnvironmentContext in DDLTask

2018-04-11 Thread Nishant Bangarwa (JIRA) 
Nishant Bangarwa created HIVE-19172:
---

 Summary: NPE due to null EnvironmentContext in DDLTask
 Key: HIVE-19172
 URL: https://issues.apache.org/jira/browse/HIVE-19172
 Project: Hive
  Issue Type: Task
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa


Stack Trace -
{code}
2018-04-11T02:52:51,386 ERROR [5f2e24bf-ac93-4977-84fe-aa2c5f674ea4 main] 
exec.DDLTask: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3539)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:392)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1987)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1667)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1414)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19171) Persist runtime statistics in metastore

2018-04-11 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-19171:
---

 Summary: Persist runtime statistics in metastore
 Key: HIVE-19171
 URL: https://issues.apache.org/jira/browse/HIVE-19171
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19170) Fix TestMiniDruidKafkaCliDriver

2018-04-11 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-19170:
---

 Summary: Fix TestMiniDruidKafkaCliDriver
 Key: HIVE-19170
 URL: https://issues.apache.org/jira/browse/HIVE-19170
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich


added in HIVE-18976 

the property key {{druid.kafka.query.files}} doesn't exists in 
testconfiguration.properties.

because of this TestMiniDruidKafkaCliDriver tries to run *all* qtests...which 
time out...and produce 
{code}
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed 
out) (batchId=252)
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66402: HIVE-19009 Retain and use runtime statistics thru out a session

2018-04-11 Thread Zoltan Haindrich 


> On April 11, 2018, 12:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SessionStatsSource.java
> > Lines 35 (patched)
> > 
> >
> > There should be a singleton for this cache which can be called from 
> > anywhere. e.g., see MaterializationsInvalidationCache

I will add that too - probably in the next patch ; but this is the "session 
level only" implementation; a static cache would be hs2 lifetime; or wider an 
even wider scope


> On April 11, 2018, 12:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java
> > Lines 34 (patched)
> > 
> >
> > Deprecated when it was first written :)
> > Any reason for having it this way.

yes...I use it as a marker that I've pending renaming to do - when I've created 
the method it was unclear what will be


> On April 11, 2018, 12:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/reexec/SessionStatsPlugin.java
> > Lines 36-40 (patched)
> > 
> >
> > I am not sure SS stats source should be linked to driver. Its lifecycle 
> > is not tied to query or driver or SS. This should be initialized once in a 
> > HS2 lifecycle.

I'll add the hs2 lifecycle level caching as a separate thing;
I think the session level may probably be usefull to reproduce problems with it 
later.

What happens right here is that: if the session doesn't yet have a 
sessionstatssource; a new one will be created - the creation might be also 
moved to sessionstate; but I didn't wanted to use the specific  implementation 
class there ; this will probably change when longer runtime stats will be 
retained for longer period (metastore)


> On April 11, 2018, 12:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
> > Lines 123 (patched)
> > 
> >
> > Why is stored in SessionState. SS is per query, whereas statscache is 
> > global.

SessionState is per session - at least currently it works like that ; will add 
the extension to hs2/persisted level in a follow up - I think it will be pretty 
straight forward...

but I will probably rethink how the caching level is configured


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66402/#review200829
---


On April 10, 2018, 12:31 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66402/
> ---
> 
> (Updated April 10, 2018, 12:31 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-19009
> https://issues.apache.org/jira/browse/HIVE-19009
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * retain runtime stats at session level
> * some fixes to support runtime stats during vectorized execution
> * make collection less strict; there are cases when the same op tree is 
> evaluated multiple times in a query
> * minor fixes and added some logging
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> dd94d4db87514de9050ea5669f73027ff3dd6937 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 
> 58fa5f2287e1221f300e16a78aa76dc6fc23cf47 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 75f928b69d3d7b206564216d24be450848a1fe8a 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 
> 0f6a80ef0d7b768371a932e5bda75348955d0369 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 13a2fc478fda244eca8632d12f2d8f46ae280a63 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java 
> 57762edbdfbf4f77b051f42faf313db882878969 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SessionStatsSource.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java 
> a4cb6e977190c4f5fa275619a8e97755ad1f55d8 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSources.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java 
> 2b0d23c6f237915684d59856cb62001c01c8209c 
>   ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java 
> 93031712dc6fb60a6d618c8754e50def489a12af 
>   ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java 
> 4ee3c14b3988521b3f44d02f032163734ed9e4d9 
>   ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java 
> 707858700f4d06301fb873fb47f8d9d9b101d056 
>

Re: contributor role

2018-04-11 Thread Antal Sinkovits 
Thanks for the clarification Thejas,

I was under the impression, that it needs some additional manual step. But
its working fine.

Regards,
Antal

On Tue, Apr 10, 2018 at 10:39 PM, Thejas Nair  wrote:

> Hi Antal,
> Are you referring to contributor role in Jira ?
> I think you should be able to assign jiras to yourself without that.
> ie, anyone with jira account should be equivalent to a contributor.
> Are you facing any issues with that ?
>
> Thanks,
> Thejas
>
>
> On Tue, Apr 10, 2018 at 9:52 AM, Antal Sinkovits
>  wrote:
> > Hi Team,
> >
> > Could you please assign contributor role to me. My jira username is
> > asinkovits.
> > Thanks.
> >
> > Regards,
> > Antal
>

[jira] [Created] (HIVE-19169) llap: Timed out after 90 secs

2018-04-11 Thread Zoltan Haindrich (JIRA) 
Zoltan Haindrich created HIVE-19169:
---

 Summary: llap: Timed out after 90 secs
 Key: HIVE-19169
 URL: https://issues.apache.org/jira/browse/HIVE-19169
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Zoltan Haindrich


a more or less recent hive/master with tpcds1000; while running only 1 query at 
a time on the cluster; sometimes a "timeout" happens - this could be even a 
misconfiguration problem...I'm not sure I've set it up correctly.

what I see from: it seems like the attempt have entered the queue and accepted 
- after almost 90 seconds there are messages that it will be pre-empted - and 
the 90 sec timeout happens.

hive.log; example: attempt_1522319554594_0065_19_05_000119_14 failed
{code}
2018-04-10T13:20:25,178 ERROR [HiveServer2-Background-Pool: Thread-4194]: 
SessionState (SessionState.java:printError(1214)) - Vertex failed, 
vertexName=Reducer 3, vertexId=vertex_1522319554594_0065_19_05, 
diagnostics=[Task failed, taskId=task_1522319554594_0065_19_05_000119, 
diagnostics=[TaskAttempt 0 killed, TaskAttempt 1 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_1 Timed out after 90 
secs], TaskAttempt 2 killed, TaskAttempt 3 killed, TaskAttempt 4 killed, 
TaskAttempt 5 killed, TaskAttempt 6 killed, TaskAttempt 7 killed, TaskAttempt 8 
killed, TaskAttempt 9 killed, TaskAttempt 10 killed, TaskAttempt 11 killed, 
TaskAttempt 12 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_12 Timed out after 90 
secs], TaskAttempt 13 killed, TaskAttempt 14 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_14 Timed out after 90 
secs], TaskAttempt 15 failed, 
info=[org.apache.hadoop.ipc.RemoteException(java.lang.RuntimeException): 
attempt_1522319554594_0065_19_08_18_6 was not registered and couldn't be 
removed
2018-04-10T13:20:25,260 ERROR [HiveServer2-Background-Pool: Thread-4194]: 
ql.Driver (SessionState.java:printError(1214)) - FAILED: Execution Error, 
return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, 
vertexName=Reducer 3, vertexId=vertex_1522319554594_0065_19_05, 
diagnostics=[Task failed, taskId=task_1522319554594_0065_19_05_000119, 
diagnostics=[TaskAttempt 0 killed, TaskAttempt 1 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_1 Timed out after 90 
secs], TaskAttempt 2 killed, TaskAttempt 3 killed, TaskAttempt 4 killed, 
TaskAttempt 5 killed, TaskAttempt 6 killed, TaskAttempt 7 killed, TaskAttempt 8 
killed, TaskAttempt 9 killed, TaskAttempt 10 killed, TaskAttempt 11 killed, 
TaskAttempt 12 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_12 Timed out after 90 
secs], TaskAttempt 13 killed, TaskAttempt 14 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_14 Timed out after 90 
secs], TaskAttempt 15 failed, 
info=[org.apache.hadoop.ipc.RemoteException(java.lang.RuntimeException): 
attempt_1522319554594_0065_19_08_18_6 was not registered and couldn't be 
removed
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: 
FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 
3, vertexId=vertex_1522319554594_0065_19_05, diagnostics=[Task failed, 
taskId=task_1522319554594_0065_19_05_000119, diagnostics=[TaskAttempt 0 killed, 
TaskAttempt 1 failed, info=[AttemptID:attempt_1522319554594_0065_19_05_000119_1 
Timed out after 90 secs], TaskAttempt 2 killed, TaskAttempt 3 killed, 
TaskAttempt 4 killed, TaskAttempt 5 killed, TaskAttempt 6 killed, TaskAttempt 7 
killed, TaskAttempt 8 killed, TaskAttempt 9 killed, TaskAttempt 10 killed, 
TaskAttempt 11 killed, TaskAttempt 12 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_12 Timed out after 90 
secs], TaskAttempt 13 killed, TaskAttempt 14 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_14 Timed out after 90 
secs], TaskAttempt 15 failed, 
info=[org.apache.hadoop.ipc.RemoteException(java.lang.RuntimeException): 
attempt_1522319554594_0065_19_08_18_6 was not registered and couldn't be 
removed
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed, 
vertexName=Reducer 3, vertexId=vertex_1522319554594_0065_19_05, 
diagnostics=[Task failed, taskId=task_1522319554594_0065_19_05_000119, 
diagnostics=[TaskAttempt 0 killed, TaskAttempt 1 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_1 Timed out after 90 
secs], TaskAttempt 2 killed, TaskAttempt 3 killed, TaskAttempt 4 killed, 
TaskAttempt 5 killed, TaskAttempt 6 killed, TaskAttempt 7 killed, TaskAttempt 8 
killed, TaskAttempt 9 killed, TaskAttempt 10 killed, TaskAttempt 11 killed, 
TaskAttempt 12 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_12 Timed out after 90 
secs], TaskAttempt 13 killed, TaskAttempt 14 failed, 
info=[AttemptID:attempt_1522319554594_0065_19_05_000119_14

[jira] [Created] (HIVE-19168) Ranger changes for llap commands

2018-04-11 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-19168:


 Summary: Ranger changes for llap commands
 Key: HIVE-19168
 URL: https://issues.apache.org/jira/browse/HIVE-19168
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 3.0.0, 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


New llap commands "llap cluster -info" and "llap cache -purge" require some 
changes so that Ranger can log the commands for auditing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19167) Map data type doesn't keep the order of the key/values pairs as read (Part 2, The Sequel or SQL)

2018-04-11 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-19167:
---

 Summary: Map data type doesn't keep the order of the key/values 
pairs as read (Part 2, The Sequel or SQL)   
 Key: HIVE-19167
 URL: https://issues.apache.org/jira/browse/HIVE-19167
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline
 Fix For: 3.1.0


HIVE-19116: "Vectorization: Vector Map data type doesn't keep the order of the 
key/values pairs as read" didn't fix all the places where HashMap is used 
instead of LinkedHashMap.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)