[jira] [Created] (HIVE-16325) Tez session refresh based on a time interval fails

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16325:
-

 Summary: Tez session refresh based on a time interval fails
 Key: HIVE-16325
 URL: https://issues.apache.org/jira/browse/HIVE-16325
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


TezSessionSTate setup requires access to SessionState (which is a little 
absurd).
The expire / renew threads are not setup with any SessionState.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] split metastore and service

[Thejas Nair]

Also, thanks for the email thread to bring peoples attention to this change.

On Tue, Mar 28, 2017 at 6:35 PM, Thejas Nair  wrote:

> +1
> Thanks for looking into this!
>
>
> On Tue, Mar 28, 2017 at 11:26 AM, Eugene Koifman  > wrote:
>
>> +1 reduce the number of uber jars
>>
>>
>> On 3/27/17, 1:05 PM, "Sergey Shelukhin"  wrote:
>>
>> Splitting the metastore would also allow us to get rid of compile time
>> dependencies that are resolved via reflection right now.
>> +1 on the feature
>>
>> On 17/3/27, 07:33, "Zoltan Haindrich" 
>> wrote:
>>
>> >Hello,
>> >
>> >Currently the jdbc driver contains lots of hive code; which are not
>> >needed for the driver to function properly - jdbc-standalone is
>> currently
>> >a 60M binary! :)
>> >
>> >I've opened a ticket, to explore the possibilites what can be done in
>> >this aspect to reduce jdbc's dependencies.
>> >
>> >I was able to remove most of the service and the metastore
>> dependencies -
>> >by introducing 2 new modules: I called them metastore-api and
>> >service-client.
>> >As a change like this would mean that the released jars name and
>> purpose
>> >would change - I didn't wanted to just file a jira about it :)
>> >
>> >So...I would like to ask for opinions or any concerns against doing
>> the
>> >following:
>> >
>> >1) Splitting the metastore module; the new module would be named as
>> >metastore-X (my proposals for X are: client,rpc,if or api).
>> >  * the dependency would contain the thrift interface
>> >  * and possibly a few other source files which are needed to use it.
>> >
>> >2) Splitting the service module; the new module would be named
>> service-X
>> >(my propsal for X would be client)
>> >  * the module would contain auth related classes
>> >  * some other basic stuffs like RowSet
>> >  * connected change: jdbc driver would change the support of
>> embedded
>> >mode to only make it usable if 'service' is loaded onto the classpath
>> >
>> >With these two modules available, the size of the jdbc driver have
>> >dropped to about 21M.
>> >
>> >more info:
>> >https://issues.apache.org/jira/browse/HIVE-16214
>> >
>> >regards,
>> >Zoltan
>>
>>
>>
>>
>

Re: [DISCUSS] split metastore and service

[Thejas Nair]

+1
Thanks for looking into this!


On Tue, Mar 28, 2017 at 11:26 AM, Eugene Koifman 
wrote:

> +1 reduce the number of uber jars
>
>
> On 3/27/17, 1:05 PM, "Sergey Shelukhin"  wrote:
>
> Splitting the metastore would also allow us to get rid of compile time
> dependencies that are resolved via reflection right now.
> +1 on the feature
>
> On 17/3/27, 07:33, "Zoltan Haindrich" 
> wrote:
>
> >Hello,
> >
> >Currently the jdbc driver contains lots of hive code; which are not
> >needed for the driver to function properly - jdbc-standalone is
> currently
> >a 60M binary! :)
> >
> >I've opened a ticket, to explore the possibilites what can be done in
> >this aspect to reduce jdbc's dependencies.
> >
> >I was able to remove most of the service and the metastore
> dependencies -
> >by introducing 2 new modules: I called them metastore-api and
> >service-client.
> >As a change like this would mean that the released jars name and
> purpose
> >would change - I didn't wanted to just file a jira about it :)
> >
> >So...I would like to ask for opinions or any concerns against doing
> the
> >following:
> >
> >1) Splitting the metastore module; the new module would be named as
> >metastore-X (my proposals for X are: client,rpc,if or api).
> >  * the dependency would contain the thrift interface
> >  * and possibly a few other source files which are needed to use it.
> >
> >2) Splitting the service module; the new module would be named
> service-X
> >(my propsal for X would be client)
> >  * the module would contain auth related classes
> >  * some other basic stuffs like RowSet
> >  * connected change: jdbc driver would change the support of embedded
> >mode to only make it usable if 'service' is loaded onto the classpath
> >
> >With these two modules available, the size of the jdbc driver have
> >dropped to about 21M.
> >
> >more info:
> >https://issues.apache.org/jira/browse/HIVE-16214
> >
> >regards,
> >Zoltan
>
>
>
>

[jira] [Created] (HIVE-16324) Truncate table should not work when EXTERNAL property of table is true

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-16324:
--

 Summary: Truncate table should not work when EXTERNAL property of 
table is true
 Key: HIVE-16324
 URL: https://issues.apache.org/jira/browse/HIVE-16324
 Project: Hive
  Issue Type: Bug
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Currently if you create an external table using the command {{CREATE EXTERNAL 
TABLE table_name}} the {{TRUNCATE table table_name}} command fails as expected 
because only managed tables should be allowed to be truncated.

But if you set the external property of a previously managed table using 
{{ALTER TABLE table_name SET TBLPROPERTIES('EXTERNAL'='true')}}, truncate table 
command does not object and deletes all the data from the external table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16323) HS2 JDOPersistenceManagerFactory.pmCache leaks after HIVE-14204

[Daniel Dai (JIRA)]

Daniel Dai created HIVE-16323:
-

 Summary: HS2 JDOPersistenceManagerFactory.pmCache leaks after 
HIVE-14204
 Key: HIVE-16323
 URL: https://issues.apache.org/jira/browse/HIVE-16323
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Daniel Dai
Assignee: Daniel Dai


Hive.loadDynamicPartitions creates threads with new embedded rawstore, but 
never close them, thus we leak PersistenceManager one per such thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16322) Create a TestNegativeEncryptedCliDriver for negative tests

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-16322:
--

 Summary: Create a TestNegativeEncryptedCliDriver for negative tests
 Key: HIVE-16322
 URL: https://issues.apache.org/jira/browse/HIVE-16322
 Project: Hive
  Issue Type: Test
  Components: Hive
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Currently there is no negative test CLI driver for running encrypted tests. The 
current mechanism to test negative tests for encrypted q-file tests is to set 
hive.cli.ignore.error=true and let the error occur and compared in q.out file. 
we should create a negative CLI driver just like TestNegativeCliDriver for 
encrypted use-cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16321) Possible deadlock in metastore with Acid workload

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-16321:
-

 Summary: Possible deadlock in metastore with Acid workload
 Key: HIVE-16321
 URL: https://issues.apache.org/jira/browse/HIVE-16321
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.3.0
Reporter: Eugene Koifman


TxnStore.MutexAPI is a mechanism how different Metastore instances can 
coordinate their operations.  It uses a JDBCConnection to achieve it.

In some cases this may lead to deadlock.  TxnHandler uses a connection pool of 
fixed size.  Suppose you X simultaneous calls to  TxnHandlerlock(), where X is 
>= size of the pool.  This take all connections form the pool, so when
{noformat}
handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
{noformat} 
is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the 
pool is empty and the system is deadlocked.

MutexAPI can't use the same connection as the operation it's protecting.  
(TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).

We could make MutexAPI use a separate connection pool (size > 'primary' conn 
pool).

Or we could make TxnHandler.lock(LockRequest rqst) return immediately after 
enqueueing the lock with the expectation that the caller will always follow up 
with a call to checkLock(CheckLockRequest rqst).

cc [~f1sherox]





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 58004: HIVE-16228: Support for subqueries in complex expressions in SELECT

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58004/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16228
https://issues.apache.org/jira/browse/HIVE-16228


Repository: hive-git


Description
---

This patch continues the work done in HIVE-16091 to add support for subqueries 
in SELECT for complex expressions e.g.
select 1+  from table

Limitation
* As mentioned in JIRA this patch doesn't have support for subqueries in UDAs 
and UDTFs


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 7c96f3d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e75e892 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 0ec24b5 
  ql/src/test/queries/clientnegative/subquery_select_complex_expr.q 8e6cf66 
  ql/src/test/queries/clientnegative/subquery_select_udf.q a52a5a9 
  ql/src/test/queries/clientpositive/perf/query9.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_select.q cb7d7aa 
  ql/src/test/results/clientnegative/subquery_in_select.q.out a8e1b99 
  ql/src/test/results/clientnegative/subquery_select_aggregate.q.out a8e1b99 
  ql/src/test/results/clientnegative/subquery_select_complex_expr.q.out a8e1b99 
  ql/src/test/results/clientnegative/subquery_select_udf.q.out a8e1b99 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 6fb253b 
  ql/src/test/results/clientpositive/perf/query9.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/58004/diff/1/


Testing
---

Added more positive and negative test cases
Pre-commit testing


Thanks,

Vineet Garg

[jira] [Created] (HIVE-16320) Flaky test: TestBeeLineDriver.testCliDriver[escape_comments]

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16320:
---

 Summary: Flaky test: 
TestBeeLineDriver.testCliDriver[escape_comments]
 Key: HIVE-16320
 URL: https://issues.apache.org/jira/browse/HIVE-16320
 Project: Hive
  Issue Type: Sub-task
Reporter: Sahil Takiar






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16319) LLAP: Better handling of an empty wait queue, should try scheduling checks

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-16319:
-

 Summary: LLAP: Better handling of an empty wait queue, should try 
scheduling checks
 Key: HIVE-16319
 URL: https://issues.apache.org/jira/browse/HIVE-16319
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16318) LLAP cache: address some issues in 2.2/2.3

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-16318:
---

 Summary: LLAP cache: address some issues in 2.2/2.3
 Key: HIVE-16318
 URL: https://issues.apache.org/jira/browse/HIVE-16318
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


We've run into HIVE-16233 and HIVE-15665 and given that 2.2 and 2.3 releases 
are approaching we are going to add workarounds for them, and then commit the 
above patches and revert the workarounds as soon as we can.
Unfortunately this will result in cache wasting some memory on some datasets, 
but the alternatives, when they are encountered (usually only on large 
datasets), are worse.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 58001: HIVE-16289 add hints for semijoin reduction

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58001/
---

Review request for hive.


Repository: hive-git


Description
---

see jira


Diffs
-

  itests/src/test/resources/testconfiguration.properties 3c98a57 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 b9f5912 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 73a9b0f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e75e892 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g 8e70a46 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java ec76fb7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f2a6ade 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
18e4fbd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bcf3691 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 71c7310 
  ql/src/test/queries/clientpositive/semijoin_hint.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/58001/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 58001: HIVE-16289 add hints for semijoin reduction

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58001/
---

Review request for hive.


Repository: hive-git


Description
---

see jira


Diffs
-

  itests/src/test/resources/testconfiguration.properties 3c98a57 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 b9f5912 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
 73a9b0f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e75e892 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HintParser.g 8e70a46 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java ec76fb7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f2a6ade 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
18e4fbd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bcf3691 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 71c7310 
  ql/src/test/queries/clientpositive/semijoin_hint.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/semijoin_hint.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/58001/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

Re: [DISCUSS] split metastore and service

[Eugene Koifman]

+1 reduce the number of uber jars


On 3/27/17, 1:05 PM, "Sergey Shelukhin"  wrote:

Splitting the metastore would also allow us to get rid of compile time
dependencies that are resolved via reflection right now.
+1 on the feature

On 17/3/27, 07:33, "Zoltan Haindrich"  wrote:

>Hello,
>
>Currently the jdbc driver contains lots of hive code; which are not
>needed for the driver to function properly - jdbc-standalone is currently
>a 60M binary! :)
>
>I've opened a ticket, to explore the possibilites what can be done in
>this aspect to reduce jdbc's dependencies.
>
>I was able to remove most of the service and the metastore dependencies -
>by introducing 2 new modules: I called them metastore-api and
>service-client.
>As a change like this would mean that the released jars name and purpose
>would change - I didn't wanted to just file a jira about it :)
>
>So...I would like to ask for opinions or any concerns against doing the
>following:
>
>1) Splitting the metastore module; the new module would be named as
>metastore-X (my proposals for X are: client,rpc,if or api).
>  * the dependency would contain the thrift interface
>  * and possibly a few other source files which are needed to use it.
>
>2) Splitting the service module; the new module would be named service-X
>(my propsal for X would be client)
>  * the module would contain auth related classes
>  * some other basic stuffs like RowSet
>  * connected change: jdbc driver would change the support of embedded
>mode to only make it usable if 'service' is loaded onto the classpath
>
>With these two modules available, the size of the jdbc driver have
>dropped to about 21M.
>
>more info:
>https://issues.apache.org/jira/browse/HIVE-16214
>
>regards,
>Zoltan

[jira] [Created] (HIVE-16317) CASE .. NULL in JOIN condition can trigger SemanticException

[Remus Rusanu (JIRA)]

Remus Rusanu created HIVE-16317:
---

 Summary: CASE .. NULL in JOIN condition can trigger 
SemanticException
 Key: HIVE-16317
 URL: https://issues.apache.org/jira/browse/HIVE-16317
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Remus Rusanu
Assignee: Remus Rusanu


{noformat}
hive> explain select a.key from src a join src b on case when a.key=b.key then 
1 else null end;
FAILED: SemanticException tok_null encountered with 0 children
{noformat}

{noformat}
"2489e686-bf69-4f37-b733-72491351644b main"
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2629)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2904)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2808)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:8916)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10506)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10419)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:408)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at 
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:511)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at 
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}

In HIVE-15708 this causes query31 to fail CBO optimization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57626: HIVE-16164: Provide mechanism for passing HMS notification ID between transactional and non-transactional listeners.

[Sergio Pena]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57626/
---

(Updated March 28, 2017, 3:17 p.m.)


Review request for hive.


Bugs: HIVE-16164
https://issues.apache.org/jira/browse/HIVE-16164


Repository: hive-git


Description
---

This fix updates the EnvironmentContext with a DB_NOTIFICATION_EVENT_ID 
property from withing the DbNotificationListener class. It then passes the 
EnvironmentContext from transactional listeners to non-transactional listeners 
so that the eventId is shared between them.

The patch provides the following changes:
- DbNotificationListener   Changes to pass the EnvironmentContext from 
transactional to non-transactional listeners.
- HiveAlterHandler Changes to pass the EnvironmentContext from 
transactional to non-transactional listeners.
- MetaStoreListenerNotifierNew helper class that wraps the notification 
call to the listeners.
- TestObjectStore  Verifies that the addNotificationEvent() method 
saves the eventId on the NotificationEvent object.
- TestDbNotificationListener   Verifies that any HMS call is passing the 
DB_NOTIFICATION_EVENT_ID to non-transactional listeners.


Diffs (updated)
-

  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 ea6cb790bc03fc38be07a3be7be34aa9c5d293ba 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/MetaStoreEventListenerConstants.java
 PRE-CREATION 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 1cf47c36cb490ce0b17ffe312cd2e9fc4bb7cd9a 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
4ce6a659c94265d83abbaedb9c52d7e15bf8dde6 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
80b1e98b9efbf068f0cd32303499a3a9b5bbd67b 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreListenerNotifier.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/events/ListenerEvent.java 
62aeb8cc343fc4aab72e76da890e36ac3a16b7bf 
  metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
1f87eeb18f6edf7351b3c8da6a6826c08656e48c 


Diff: https://reviews.apache.org/r/57626/diff/6/

Changes: https://reviews.apache.org/r/57626/diff/5-6/


Testing
---

HiveQA showed only one test failure. it is fixed, and waiting for HiveQA to 
complete 100% tests.


Thanks,

Sergio Pena

[jira] [Created] (HIVE-16316) Prepare master branch for 3.0.0 development.

[Naveen Gangam (JIRA)]

Naveen Gangam created HIVE-16316:


 Summary: Prepare master branch for 3.0.0 development.
 Key: HIVE-16316
 URL: https://issues.apache.org/jira/browse/HIVE-16316
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.3.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


branch-2 is now being used for 2.3.0 development. The build files will need to 
reflect this change.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16315) Describe table doesn't show num of partitions

[Rui Li (JIRA)]

Rui Li created HIVE-16315:
-

 Summary: Describe table doesn't show num of partitions
 Key: HIVE-16315
 URL: https://issues.apache.org/jira/browse/HIVE-16315
 Project: Hive
  Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li


This doesn't comply with our wiki: 
https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-Examples



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16314) Data type change in ORC hive table

[Kirthiga Balasubramanian (JIRA)]

Kirthiga Balasubramanian created HIVE-16314:
---

 Summary: Data type change in ORC hive table
 Key: HIVE-16314
 URL: https://issues.apache.org/jira/browse/HIVE-16314
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.2.1
 Environment: Production
Reporter: Kirthiga Balasubramanian
Priority: Critical


Change in column data type of ORC hive table is throwing exception.

ORC table is partitioned. The data type of one column has been changed from 
tinyint to int. 

1) Querying the column for a partition after the data type change is giving 
results.
2)Querying the column for a partition before the data type change was made is 
throwing the below exception





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16313) Flaky test: TestBeeLineDriver[drop_with_concurrency]

[Peter Vary (JIRA)]

Peter Vary created HIVE-16313:
-

 Summary: Flaky test: TestBeeLineDriver[drop_with_concurrency]
 Key: HIVE-16313
 URL: https://issues.apache.org/jira/browse/HIVE-16313
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Peter Vary
Assignee: Peter Vary


TestBeeLineDriver test become flaky:
{code}
28a29 > Acquired the compile lock
{code}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 57944: HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57944/#review170280
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
Lines 224 (patched)


Can you elaborate a bit about this change? Maybe a comment could be left on 
why it is needed.

Further, since in L197 we check whether number of keys in child is less 
than number of keys in parent, and here we enforce greater than... is it not 
the same as changing condition in L197 to _cRSc.getKeyCols().size() != 
pRSNc.getKeyCols().size()_?


- Jesús Camacho Rodríguez


On March 27, 2017, 7:54 a.m., Remus Rusanu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57944/
> ---
> 
> (Updated March 27, 2017, 7:54 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9
> 
> 
> Diffs
> -
> 
>   druid-handler/pom.xml 926c53b 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java
>  0b35428 
>   pom.xml 4c73e27 
>   ql/pom.xml 7db0ede 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  1d78b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  0c6996c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 6ccd879 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java
>  9a65de3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java
>  2aadf50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExtractDate.java
>  4edc4df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
>  4b8568e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewFilterScanRule.java
>  38d7906 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  69e157e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  0dc0c24 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  27990a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
>  e840938 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  a05b89c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
>  2b075be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e75e892 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
>  7229cc7 
>   ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out 92f1365 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join1.q.out fe4bc4f 
>   ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.out 0bb18cc 
>   ql/src/test/results/clientpositive/constprog2.q.out 7bfd0cf 
>   ql/src/test/results/clientpositive/druid_basic2.q.out bc9410b 
>   ql/src/test/results/clientpositive/druid_intervals.q.out ca3febf 
>   ql/src/test/results/clientpositive/druid_timeseries.q.out 6b2ffe9 
>   ql/src/test/results/clientpositive/druid_topn.q.out 57d6586 
>   ql/src/test/results/clientpositive/filter_cond_pushdown.q.out 8c6f0f1 
>   ql/src/test/results/clientpositive/fouter_join_ppr.q.out 1897aa7 
>   ql/src/test/results/clientpositive/index_auto_unused.q.out 8a270d6 
>   ql/src/test/results/clientpositive/join45.q.out 249fe9c 
>   ql/src/test/results/clientpositive/join_alt_syntax.q.out cef5272 
>   ql/src/test/results/clientpositive/join_cond_pushdown_1.q.out 9461a1b 
>   ql/src/test/results/clientpositive/join_cond_pushdown_3.q.out 2205c4e 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual1.q.out 966330d 
>   ql/src/test/results/clientpositive/join_cond_pushdown_unqual3.q.out 54a2d06 
>   ql/src/test/results/clientpositive/join_merging.q.out 79a7023 
>   ql/src/test/results/clientpositive/llap/auto_smb_mapjoin_14.q.out a867bd2 
>   ql/src/test/results/clientpositive/llap/auto_sortmerge_join_9.q.out c08a534 
>   ql/src/test/results/clientpositive/llap/constprog_semijoin.q.out 98d2328 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_2.q.out 
> d291e7d 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out e806ef3 
>   ql/src/test/results/clientpositive/llap/explainuser_4.q.out 9804569 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 

Re: Review Request 57944: HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9

[Jesús Camacho Rodríguez]

> On March 27, 2017, 10:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/druid_intervals.q.out
> > Line 204 (original), 205 (patched)
> > 
> >
> > Seems like we are not able to translate between filters to druid time 
> > ranges. Could be related to cast we are inserting for timestamps.

CBO might be failing, as Druid select query contains default interval + all 
dimensions and metrics, which is what we do in those cases... Worth taking a 
look.


> On March 27, 2017, 10:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/druid_intervals.q.out
> > Line 298 (original), 315 (patched)
> > 
> >
> > Not able to translate IN filter to interval range. My assumption is its 
> > related to casting we are now doing on timestamps. 
> > Worthwhile to take a look at CALCITE-1655 as well.

Same as above.


> On March 27, 2017, 10:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/druid_topn.q.out
> > Line 123 (original), 123 (patched)
> > 
> >
> > Not pushing topN queries. This is a regression, but expected. Lets 
> > create a follow-up jira for this.

As below. This is expected as semantics of custom granularity + limit different 
in Druid and SQL.


> On March 27, 2017, 10:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/druid_topn.q.out
> > Line 190 (original), 246 (patched)
> > 
> >
> > Not pushing limit with Gby on time. This is a regression, but expected. 
> > We can have this change.
> > Lets create a follow-up jira for this.

This is expected as semantics of custom granularity + limit different in Druid 
(oby + top 10 per month) and SQL (oby + top 10 among all months).


> On March 27, 2017, 10:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/druid_topn.q.out
> > Line 258 (original), 370 (patched)
> > 
> >
> > Not pushing limits with GBy + Order by. Expected Regression. Lets 
> > create a follow-up jira for this.

As above. This is expected as semantics of custom granularity + limit different 
in Druid and SQL.


- Jesús


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57944/#review170208
---


On March 27, 2017, 7:54 a.m., Remus Rusanu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57944/
> ---
> 
> (Updated March 27, 2017, 7:54 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15708 Upgrade Calcite to 1.12 and Avatica to 1.9
> 
> 
> Diffs
> -
> 
>   druid-handler/pom.xml 926c53b 
>   
> druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java
>  0b35428 
>   pom.xml 4c73e27 
>   ql/pom.xml 7db0ede 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
>  1d78b4c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java
>  0c6996c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 6ccd879 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java
>  9a65de3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java
>  2aadf50 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExtractDate.java
>  4edc4df 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterJoinRule.java
>  4b8568e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HiveMaterializedViewFilterScanRule.java
>  38d7906 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  69e157e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
>  0dc0c24 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  27990a2 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
>  e840938 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  a05b89c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
>  2b075be 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e75e892 
>   
> ql/src/te

[jira] [Created] (HIVE-16312) Flaky test: TestHCatClient.testTransportFailure

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16312:
--

 Summary: Flaky test: TestHCatClient.testTransportFailure
 Key: HIVE-16312
 URL: https://issues.apache.org/jira/browse/HIVE-16312
 Project: Hive
  Issue Type: Sub-task
Reporter: Barna Zsombor Klara


The test has been failing consistently for 10+ builds.
Error message:
{code}
Error Message

The expected exception was never thrown.
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide

[Colin Ma (JIRA)]

Colin Ma created HIVE-16311:
---

 Summary: Improve the performance for FastHiveDecimalImpl.fastDivide
 Key: HIVE-16311
 URL: https://issues.apache.org/jira/browse/HIVE-16311
 Project: Hive
  Issue Type: Improvement
Affects Versions: 2.2.0
Reporter: Colin Ma
Assignee: Colin Ma
 Fix For: 2.2.0


FastHiveDecimalImpl.fastDivide is poor performance when evaluate the expression 
as 12345.67/123.45
There are 2 points can be improved:
1. Don't always use HiveDecimal.MAX_SCALE as scale when do the 
BigDecimal.divide.
2. Get the precision for BigInteger in a fast way if possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)