[jira] [Created] (HIVE-17354) Fix "alter view" for incremental replication

2017-08-17 Thread Tao Li (JIRA) 
Tao Li created HIVE-17354:
-

 Summary: Fix "alter view" for incremental replication
 Key: HIVE-17354
 URL: https://issues.apache.org/jira/browse/HIVE-17354
 Project: Hive
  Issue Type: Bug
Reporter: Tao Li
Assignee: Tao Li






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61604: HIVE-17265

2017-08-17 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61604/
---

(Updated Aug. 18, 2017, 12:57 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17265
https://issues.apache.org/jira/browse/HIVE-17265


Repository: hive-git


Description
---

HIVE-17265


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/CompilationOpContext.java 
949f8738f8bf2a707e0b4514ef2aa5ca48ed7dd6 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
 5dff242a17eab126267c8333860fcf9dfc3b89be 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
1d49568c8cda2aa6c27568f023dc39ec20ab1949 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
66248654e8c0b7d242a089f5ac1a624dfb06598d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 6cb0559ea04abebb1da38e93c2162666cdf95af9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
d6695ccbf2611551d2038582df7faa69efd4013f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 
565fbef0aa1a63c50679a4057cb23e8c0092bc7d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/PrunedPartitionList.java 
da2e1e2d8b06d04992763288709c423a4f048b4e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
4faec05bb0f7c728453fa8bd3a5ca2a6966048d4 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 
3b9ab41bed25cc4cb0122842897fffbcd0ec15c4 


Diff: https://reviews.apache.org/r/61604/diff/2/

Changes: https://reviews.apache.org/r/61604/diff/1-2/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-17353) The ResultSets are not accessible if running multiple queries within the same HiveStatement

2017-08-17 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-17353:
---

 Summary: The ResultSets are not accessible if running multiple 
queries within the same HiveStatement 
 Key: HIVE-17353
 URL: https://issues.apache.org/jira/browse/HIVE-17353
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu


The following queries would fail,
{noformat}
ResultSet rs1 =
stmt.executeQuery("select * from testMultipleResultSets1");
ResultSet rs2 =
stmt.executeQuery("select * from testMultipleResultSets2");
rs1.next();
rs2.next();
{noformat}

with the exception:
{noformat}
[HiveServer2-Handler-Pool: Thread-208]: Error fetching results: 
org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle: 
OperationHandle [opType=EXECUTE_STATEMENT, 
getHandleIdentifier()=8a1c4fe5-e80b-4d9a-b673-78d92b3baaa8]
at 
org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:177)
at 
org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:462)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:691)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17352) HiveSever2 error with "Illegal Operation state transition from CLOSED to FINISHED"

2017-08-17 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-17352:
-

 Summary: HiveSever2 error with "Illegal Operation state transition 
from CLOSED to FINISHED"
 Key: HIVE-17352
 URL: https://issues.apache.org/jira/browse/HIVE-17352
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


HiveSever2 error with "Illegal Operation state transition from CLOSED to 
FINISHED"
Many cases like CANCELED, TIMEDOUT AND CLOSED are handled. Need to handle 
FINISHED in runQuery() method.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17351) use new slider package installation command in run.sh

2017-08-17 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17351:
---

 Summary: use new slider package installation command in run.sh
 Key: HIVE-17351
 URL: https://issues.apache.org/jira/browse/HIVE-17351
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


The old syntax does not include some perf improvements in newer versions of 
Slider.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17350) metrics errors when retrying HS2 startup

2017-08-17 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-17350:
---

 Summary: metrics errors when retrying HS2 startup
 Key: HIVE-17350
 URL: https://issues.apache.org/jira/browse/HIVE-17350
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Looks like there are some sort of retries that happen when HS2 init fails. When 
HS2 startup fails for an unrelated reason and is retried, the metrics source 
initialization fails on subsequent attempts. 
{noformat}
2017-08-15T23:31:47,650 WARN  [main]: impl.MetricsSystemImpl 
(MetricsSystemImpl.java:init(152)) - hiveserver2 metrics system already 
initialized!
2017-08-15T23:31:47,650 ERROR [main]: metastore.HiveMetaStore 
(HiveMetaStore.java:init(438)) - error in Metrics init: 
java.lang.reflect.InvocationTargetException null
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.common.metrics.common.MetricsFactory.init(MetricsFactory.java:42)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:435)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6892)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:140)
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1653)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at 
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3612)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3664)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3644)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
at 
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)
at 
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:128)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:113)
at 
org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:139)
at 
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:595)
at 
org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:97)
at 
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:843)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:712)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source 
hiveserver2 already exists!

[jira] [Created] (HIVE-17349) Enforce 1 TransactionBatch per StreamingConnection

2017-08-17 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-17349:
-

 Summary: Enforce 1 TransactionBatch per StreamingConnection
 Key: HIVE-17349
 URL: https://issues.apache.org/jira/browse/HIVE-17349
 Project: Hive
  Issue Type: Improvement
  Components: HCatalog, Transactions
Reporter: Eugene Koifman
Priority: Minor


There is a comment on _hcatalog.streaming.StreamingConnection_
{noformat}
Note: the expectation is that there is at most 1 TransactionBatch outstanding 
for any given
 * StreamingConnection.  Violating this may result in "out of sequence 
response".
{noformat}

Why not enforce this in the code...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61712: HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml

2017-08-17 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61712/#review183137
---



It is a bigger patch, you might want to run yetus to see if it has any comments 
:)

Otherwise I like your approach.
One question below.

Thanks,
Peter


metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 550 (patched)


Why not use the HashMap(Map m) constructor to use 
the defaults?


- Peter Vary


On Aug. 17, 2017, 2:27 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61712/
> ---
> 
> (Updated Aug. 17, 2017, 2:27 p.m.)
> 
> 
> Review request for hive, Eugene Koifman, Peter Vary, and Thejas Nair.
> 
> 
> Bugs: HIVE-17319
> https://issues.apache.org/jira/browse/HIVE-17319
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 897fc4efd4c541ee2da30516c933647f5d4a0af4 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
> b722af6ceb8375c76c3a393cee79ed2387555b69 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61712/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

Re: Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

2017-08-17 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/#review183136
---


Ship it!




Ship It!

- Sergio Pena


On Aug. 17, 2017, 7:56 a.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60085/
> ---
> 
> (Updated Aug. 17, 2017, 7:56 a.m.)
> 
> 
> Review request for hive, Marta Kuczora, Peter Vary, and Sergio Pena.
> 
> 
> Bugs: HIVE-14747
> https://issues.apache.org/jira/browse/HIVE-14747
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client
> 
> 
> Diffs
> -
> 
>   testutils/ptest2/conf/server-env.properties.example PRE-CREATION 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
>  2f96ad03023e9f51d44d203f34edd04183605a22 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
>  b2c61f03c5bf5f170894141848c89fc26129115a 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/Context.java
>  14984bafdd18fb7636e729cc7fbbfa349b0f043e 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/ExecutionContextConfiguration.java
>  35ddd44accf34be1f5957c6df31802ee8c8022b5 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
>  e584f9c105fa134e3e267d6c6817d441b4c6b249 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java
>  a243774e52f3f5fda4a082bb99387cf5808c307b 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
>  b97b890dfe855539de2696788327ba9b4a841ff3 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestContext.java
>  PRE-CREATION 
>   
> testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
>  848faf27af1ed8945d7013b6562bab544605e4bc 
> 
> 
> Diff: https://reviews.apache.org/r/60085/diff/6/
> 
> 
> Testing
> ---
> 
> Added unit tests for the Context class.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-17348) Remove unnecessary GenSparkUtils.java.orig file

2017-08-17 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-17348:
-

 Summary: Remove unnecessary GenSparkUtils.java.orig file
 Key: HIVE-17348
 URL: https://issues.apache.org/jira/browse/HIVE-17348
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Peter Vary
Assignee: Peter Vary


HIVE-17247 added an extra file, which is most probably not needed :)
[~stakiar]? :)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 61716: HIVE-17292 - Change TestMiniSparkOnYarnCliDriver test configuration to use the configured cores

2017-08-17 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61716/
---

Review request for hive, Rui Li and Xuefu Zhang.


Bugs: HIVE-17292
https://issues.apache.org/jira/browse/HIVE-17292


Repository: hive-git


Description
---

Currently MiniSparkShim will be able to provide only 1 executor, since the 
FairScheduler is configured, and the allocation increment is not set so the 
default 1024 is used. So only 1 executor could fit into the memory restrictions.

Changing Hadoop23Shims.java, so the MiniSparkShim will able to provide the 
requested 2 executors.

Changing QTestUtil.setSparkSession, so we will wait until every executor is 
available, not only the 1st.

Changing SparkSessionImpl.getMemoryAndCores, so we use the client provided 
paralellism in case of local spark.master too.

Regenerating golden files (numReducers, and number of files changed in the 
explain plans)

The change contains 2 golden file changes 
(spark_dynamic_partition_pruning_mapjoin_only.q.out, 
spark_dynamic_partition_pruning.q.out) which are containing other neccessary 
changes for a green run, so this patch should be regenerated after their 
corresponding jiras are solved (HIVE-17347, HIVE-17346)


Diffs
-

  data/conf/spark/standalone/hive-site.xml 989e65e 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 825f826 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 
8224ef9 
  ql/src/test/results/clientpositive/spark/constprog_semijoin.q.out be33189 
  ql/src/test/results/clientpositive/spark/index_bitmap3.q.out 31c921c 
  ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out e3b205c 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out 
356f625 
  
ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out
 bf010e7 
  
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning.q.out 
d9da762 
  
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_2.q.out
 8a794e7 
  
ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_mapjoin_only.q.out
 564a6a2 
  ql/src/test/results/clientpositive/spark/spark_explain_groupbyshuffle.q.out 
9d162c9 
  ql/src/test/results/clientpositive/spark/spark_use_ts_stats_for_mapjoin.q.out 
f92356e 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
ae6f542 


Diff: https://reviews.apache.org/r/61716/diff/1/


Testing
---


Thanks,

Peter Vary

Review Request 61712: HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml

2017-08-17 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61712/
---

Review request for hive, Eugene Koifman, Peter Vary, and Thejas Nair.


Bugs: HIVE-17319
https://issues.apache.org/jira/browse/HIVE-17319


Repository: hive-git


Description
---

HIVE-17319: Make BoneCp configurable using hive properties in hive-site.xml


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
897fc4efd4c541ee2da30516c933647f5d4a0af4 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
b722af6ceb8375c76c3a393cee79ed2387555b69 
  
metastore/src/test/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61712/diff/1/


Testing
---


Thanks,

Barna Zsombor Klara

[jira] [Created] (HIVE-17347) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is failing every time

2017-08-17 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-17347:
-

 Summary: 
TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning_mapjoin_only] is 
failing every time
 Key: HIVE-17347
 URL: https://issues.apache.org/jira/browse/HIVE-17347
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Affects Versions: 3.0.0
Reporter: Peter Vary
Assignee: Peter Vary


As [~lirui] identified there was a missing file from this patch: HIVE-17247 - 
HoS DPP: UDFs on the partition column side does not evaluate correctly



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17346) TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every time

2017-08-17 Thread Peter Vary (JIRA) 
Peter Vary created HIVE-17346:
-

 Summary: 
TestMiniSparkOnYarnCliDriver[spark_dynamic_partition_pruning] is failing every 
time
 Key: HIVE-17346
 URL: https://issues.apache.org/jira/browse/HIVE-17346
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Peter Vary
Assignee: Peter Vary


The TestMiniSparkOnYarnCliDriver.testCliDriver - 
spark_dynamic_partition_pruning is failing with this diff:
{code}
Client Execution succeeded but contained differences (error code = 1) after 
executing spark_dynamic_partition_pruning.q 
714c714
<   filterExpr: ((date = '2008-04-08') and abs(((- 
UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type: 
boolean)
---
>   filterExpr: ((date = '2008-04-08') and ds is not null) 
> (type: boolean)
717c717
< predicate: ((date = '2008-04-08') and abs(((- 
UDFToLong(concat(UDFToString(day(ds)), '0'))) + 10)) is not null) (type: 
boolean)
---
> predicate: ((date = '2008-04-08') and ds is not null) 
> (type: boolean)
749c749
<   filterExpr: abs(((- UDFToLong(concat(UDFToString(day(ds)), 
'0'))) + 10)) is not null (type: boolean)
---
>   filterExpr: ds is not null (type: boolean)
751,752c751,753
<   Filter Operator
< predicate: abs(((- UDFToLong(concat(UDFToString(day(ds)), 
'0'))) + 10)) is not null (type: boolean)
---
>   Select Operator
> expressions: ds (type: string)
> outputColumnNames: _col0
754,756c755,758
< Select Operator
<   expressions: ds (type: string)
<   outputColumnNames: _col0
---
> Reduce Output Operator
>   key expressions: abs(((- 
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
>   sort order: +
>   Map-reduce partition columns: abs(((- 
> UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
758,762d759
<   Reduce Output Operator
< key expressions: abs(((- 
UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
< sort order: +
< Map-reduce partition columns: abs(((- 
UDFToLong(concat(UDFToString(day(_col0)), '0'))) + 10)) (type: bigint)
< Statistics: Num rows: 2000 Data size: 21248 Basic 
stats: COMPLETE Column stats: NONE
767c764
<  
Output was too long and had to be truncated...
{code}

I think it is caused by:
HIVE-17148 - Incorrect result for Hive join query with COALESCE in WHERE 
condition

[~allgoodok]: Am I right? Is it an intended change and only the golden file 
regeneration is needed?

Thanks,
Peter



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17345) Invalid error message for absent function

2017-08-17 Thread Alexey Bedrintsev (JIRA) 
Alexey Bedrintsev created HIVE-17345:


 Summary: Invalid error message for absent function
 Key: HIVE-17345
 URL: https://issues.apache.org/jira/browse/HIVE-17345
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability, Hive, Parser, Query Planning, UDF
Affects Versions: 1.2.1
 Environment: hive --version
Hive 1.2.1000.2.6.1.0-129
Subversion 
git://c66-slave-38a0b1a6-3/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
 -r 4c6995c87f30906625e01c28e5e111a53c2e582f
Compiled by jenkins on Wed May 31 03:19:31 UTC 2017
>From source with checksum e3ee749ba0a797bb10a7e12265daa82d
Reporter: Alexey Bedrintsev


For test purposes let have any table, e.g. like this one with two columns:
{code:sql}
create table test (x int, y int);
insert into test values (0, 0), (1, 0), (2, 0), (0, 1), (1, 1), (1, 2);
{code}

If we try to query using not existent aggregate function QQQ, then get 
descriptive error message:
{code:sql}
SELECT QQQ(y)
FROM test;
{code}
{code}
FAILED: SemanticException [Error 10011]: Line 1:7 Invalid function 'QQQ'
{code}

But if I use this function with _group by_ clause, I get confusing message 
about wrong query structure:

{code:sql}
SELECT x,
   QQQ(y)
FROM test
GROUP BY x;
{code}
{code}
FAILED: SemanticException [Error 10025]: Line 1:10 Expression not in GROUP BY 
key 'y'
{code}

For queries with _having_ clauses error message is more confusing. It is 
regarding to zero value which is absolutely correct:
{code}
SELECT x,
   QQQ(y) AS q
FROM test
GROUP BY x HAVING q > 0;
{code}
{code}
FAILED: SemanticException [Error 10025]: Line 4:7 Expression not in GROUP BY 
key '0'
{code}

*Expected result*: the same error message {code} "FAILED: SemanticException 
[Error 10011]: Line 1:7 Invalid function 'QQQ'"{code} for all listed queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17344) LocalCache element memory usage is not calculated properly.

2017-08-17 Thread Janos Gub (JIRA) 
Janos Gub created HIVE-17344:


 Summary: LocalCache element memory usage is not calculated 
properly.
 Key: HIVE-17344
 URL: https://issues.apache.org/jira/browse/HIVE-17344
 Project: Hive
  Issue Type: Bug
Reporter: Janos Gub
Assignee: Janos Gub


Orc footer cache has a calculation of memory usage:
{code:java}
public int getMemoryUsage() {
  return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads 
(semi-arbitrary).
}
{code}

ByteBuffer.remaining returns the remaining space in the bytebuffer, thus 
allowing this cache to grow until MAXSIZE/100. I think the correct solution 
would be bb.capacity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60085: HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client

2017-08-17 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60085/
---

(Updated Aug. 17, 2017, 7:56 a.m.)


Review request for hive, Marta Kuczora, Peter Vary, and Sergio Pena.


Changes
---

Added server-env.properties.example.


Bugs: HIVE-14747
https://issues.apache.org/jira/browse/HIVE-14747


Repository: hive-git


Description
---

HIVE-14747: Remove JAVA paths from profiles by sending them from ptest-client


Diffs (updated)
-

  testutils/ptest2/conf/server-env.properties.example PRE-CREATION 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
 2f96ad03023e9f51d44d203f34edd04183605a22 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
 b2c61f03c5bf5f170894141848c89fc26129115a 
  testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
1cdfdb309acd8282e593abd7ed10c87721926c60 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/Context.java
 14984bafdd18fb7636e729cc7fbbfa349b0f043e 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/ExecutionContextConfiguration.java
 35ddd44accf34be1f5957c6df31802ee8c8022b5 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
 e584f9c105fa134e3e267d6c6817d441b4c6b249 
  
testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestParser.java
 a243774e52f3f5fda4a082bb99387cf5808c307b 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
 b97b890dfe855539de2696788327ba9b4a841ff3 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestContext.java
 PRE-CREATION 
  
testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/conf/TestTestConfiguration.java
 848faf27af1ed8945d7013b6562bab544605e4bc 


Diff: https://reviews.apache.org/r/60085/diff/6/

Changes: https://reviews.apache.org/r/60085/diff/5-6/


Testing
---

Added unit tests for the Context class.


Thanks,

Barna Zsombor Klara