[jira] [Commented] (HIVE-6363) IllegalArgumentException is thrown instead of SQLException

2014-10-08 Thread JIRA

[ 
https://issues.apache.org/jira/browse/HIVE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164827#comment-14164827
 ] 

Rémy DUBOIS commented on HIVE-6363:
---

Hello everyone,

I don't understand clearly the status of this issue. I understand a patch has 
been created, but I also see that HIVE-4194 should fix this issue, but has not 
been closed after it's been reopened.
Can you please clarify?

Thanks in advance,

> IllegalArgumentException is thrown instead of SQLException
> --
>
> Key: HIVE-6363
> URL: https://issues.apache.org/jira/browse/HIVE-6363
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: chandra sekhar gunturi
>Priority: Critical
>  Labels: JDBC, unit-test
> Attachments: HIVE-6363.patch.1.txt, HIVE-6363.patch.2.txt
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> parseURL in the following code is throwing IllegalArgumentException 
> http://svn.apache.org/viewvc/hive/trunk/jdbc/src/java/org/apache/hive/jdbc/Utils.java?view=markup
>  
> This is going to break other JDBC based connectors because 
> java.sql.DriverManager doesn’t catch IllegalArgumentException while probing 
> for correct Driver for a given URL. 
> A simple test case can have class.forName(org.apache.hive.jdbc.HiveDriver) 
> (Loading hiveserver2 JDBC driver) followed by 
> class.forName(org.apache.hadoop.hive.jdbc.HiveDriver)(Loading hiveserver JDBC 
> driver).
> In this case hiveserver connection will fail with BAD URL format for 
> hiveserver. If you reverse the driver loading to hiveserver followed by 
> hiveserver2, both the connections will be successful.
> Following code in java.sql.DriverManager is causing the issue 
> [[ 
> // Worker method called by the public getConnection() methods. 
> private static Connection getConnection( 
> // Walk through the loaded registeredDrivers attempting to make a connection. 
> // Remember the first exception that gets raised so we can reraise it. 
> for(DriverInfo aDriver : registeredDrivers) { 
> // If the caller does not have permission to load the driver then 
> // skip it. 
> if(isDriverAllowed(aDriver.driver, callerCL)) { 
> try { 
> Connection con = aDriver.driver.connect(url, info); 
> if (con != null) { 
> // Success! 
> println("getConnection returning " + aDriver.driver.getClass().getName()); 
> return (con); 
> } 
> } catch (SQLException ex) { 
> if (reason == null) { 
> reason = ex; 
> } 
> } 
> } else { 
> println(" skipping: " + aDriver.getClass().getName()); 
> } 
> } 
> } 
> ]] 
> Marking it as critical because this is going to restrict consuming JDBC 
> driver in production environment where many drivers are loaded on requirement 
> rather than statically loading all drivers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Review Request 26483: Correctly handle invalid distinct queries.

2014-10-08 Thread Ashutosh Chauhan

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26483/
---

Review request for hive and John Pullokkaran.


Bugs: HIVE-8413
https://issues.apache.org/jira/browse/HIVE-8413


Repository: hive-git


Description
---

Correctly handle invalid distinct queries.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 19110ce 

Diff: https://reviews.apache.org/r/26483/diff/


Testing
---

NegativeCliDriver::invalid_distinct1.q,invalid_distinct3.q


Thanks,

Ashutosh Chauhan



[jira] [Updated] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct in incorrect context

2014-10-08 Thread Ashutosh Chauhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8413:
---
Status: Patch Available  (was: Open)

> [CBO] Handle ill-formed queries which have distinct in incorrect context 
> -
>
> Key: HIVE-8413
> URL: https://issues.apache.org/jira/browse/HIVE-8413
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8413.patch
>
>
> e.g., select hash (distinct key) from src;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct in incorrect context

2014-10-08 Thread Ashutosh Chauhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8413:
---
Attachment: HIVE-8413.patch

Correctly fail to compile in such erroneous case.

> [CBO] Handle ill-formed queries which have distinct in incorrect context 
> -
>
> Key: HIVE-8413
> URL: https://issues.apache.org/jira/browse/HIVE-8413
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 0.14.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8413.patch
>
>
> e.g., select hash (distinct key) from src;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HIVE-8413) [CBO] Handle ill-formed queries which have distinct in incorrect context

2014-10-08 Thread Ashutosh Chauhan (JIRA)
Ashutosh Chauhan created HIVE-8413:
--

 Summary: [CBO] Handle ill-formed queries which have distinct in 
incorrect context 
 Key: HIVE-8413
 URL: https://issues.apache.org/jira/browse/HIVE-8413
 Project: Hive
  Issue Type: Bug
  Components: CBO
Affects Versions: 0.14.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


e.g., select hash (distinct key) from src;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8376) Umbrella Jira for HiveServer2 dynamic service discovery

2014-10-08 Thread Vaibhav Gumashta (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8376:
---
Attachment: HiveServer2DynamicServiceDiscovery.pdf

Attaching documentation of current work, with steps to achieve rolling upgrade 
for HiveServer2 using this feature.

cc [~thejas] [~leftylev]

> Umbrella Jira for HiveServer2 dynamic service discovery
> ---
>
> Key: HIVE-8376
> URL: https://issues.apache.org/jira/browse/HIVE-8376
> Project: Hive
>  Issue Type: New Feature
>  Components: HiveServer2, JDBC
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HiveServer2DynamicServiceDiscovery.pdf
>
>
> Creating an ☂ Jira for documentation purpose. I'll add a detailed doc for the 
> implementation & usage here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8288) HiveServer2 dynamic discovery should create znodes organized by version number & add support for removing server uri's of a particular version from the server script.

2014-10-08 Thread Vaibhav Gumashta (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8288:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Patch committed to trunk and 14. Thanks for reviewing [~thejas]!

> HiveServer2 dynamic discovery should create znodes organized by version 
> number & add support for removing server uri's of a particular version from 
> the server script.
> --
>
> Key: HIVE-8288
> URL: https://issues.apache.org/jira/browse/HIVE-8288
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-8288.1.patch, HIVE-8288.2.patch, HIVE-8288.3.patch, 
> HIVE-8288.4.patch, HIVE-8288.5.patch
>
>
> Something like: 
> /hiveserver2/serverUri=host:port;version=versionInfo;sequence=sequenceNumber 
> would be better to support admin actions like removing all znodes for a 
> particular version. Also need to add an option to deregister all HiveServer2 
> instances from dynamic service discovery via command line. Something like: 
> hive --service hiveserver2 --deregister 0.14.0-SNAPSHOT



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HIVE-8412) Make reduce side join work for all join queries [Spark Branch]

2014-10-08 Thread Xuefu Zhang (JIRA)
Xuefu Zhang created HIVE-8412:
-

 Summary: Make reduce side join work for all join queries [Spark 
Branch]
 Key: HIVE-8412
 URL: https://issues.apache.org/jira/browse/HIVE-8412
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Xuefu Zhang


Regardless all these join related optimizations such as map join, bucket join, 
skewed join, etc, reduce side join is the fallback. That means, if a join query 
wasn't taken care of by any of the optimization, it should work with reduce 
side join (might in a less optimal fashion).

It's found that this isn't case at the moment. For instance, 
auto_sortmerge_join_1.q failed to execute on Spark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8288) HiveServer2 dynamic discovery should create znodes organized by version number & add support for removing server uri's of a particular version from the server script.

2014-10-08 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164709#comment-14164709
 ] 

Hive QA commented on HIVE-8288:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12673671/HIVE-8288.5.patch

{color:green}SUCCESS:{color} +1 6530 tests passed

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1183/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1183/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1183/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12673671

> HiveServer2 dynamic discovery should create znodes organized by version 
> number & add support for removing server uri's of a particular version from 
> the server script.
> --
>
> Key: HIVE-8288
> URL: https://issues.apache.org/jira/browse/HIVE-8288
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 0.14.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Critical
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-8288.1.patch, HIVE-8288.2.patch, HIVE-8288.3.patch, 
> HIVE-8288.4.patch, HIVE-8288.5.patch
>
>
> Something like: 
> /hiveserver2/serverUri=host:port;version=versionInfo;sequence=sequenceNumber 
> would be better to support admin actions like removing all znodes for a 
> particular version. Also need to add an option to deregister all HiveServer2 
> instances from dynamic service discovery via command line. Something like: 
> hive --service hiveserver2 --deregister 0.14.0-SNAPSHOT



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8222) CBO Trunk Merge: Fix Check Style issues

2014-10-08 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164698#comment-14164698
 ] 

Brock Noland commented on HIVE-8222:


[~jpullokkaran] - we agreed to let this wait because the merge patch was large, 
however I felt the consensus was on that JIRA that this would be resolved 
quickly after merge. I think should be addressed before the 0.14 release.

> CBO Trunk Merge: Fix Check Style issues
> ---
>
> Key: HIVE-8222
> URL: https://issues.apache.org/jira/browse/HIVE-8222
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8222.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8222) CBO Trunk Merge: Fix Check Style issues

2014-10-08 Thread Brock Noland (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-8222:
---
Status: Patch Available  (was: Open)

> CBO Trunk Merge: Fix Check Style issues
> ---
>
> Key: HIVE-8222
> URL: https://issues.apache.org/jira/browse/HIVE-8222
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8222.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large

2014-10-08 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164696#comment-14164696
 ] 

Brock Noland commented on HIVE-8341:


I am not sure what the distribution of the numbers are in this list but I 
assume encoding would help more than compression?

> Transaction information in config file can grow excessively large
> -
>
> Key: HIVE-8341
> URL: https://issues.apache.org/jira/browse/HIVE-8341
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8341.patch
>
>
> In our testing we have seen cases where the transaction list grows very 
> large.  We need a more efficient way of communicating the list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HIVE-8406) Research on skewed join [Spark Branch]

2014-10-08 Thread Rui Li (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li reassigned HIVE-8406:


Assignee: Rui Li

> Research on skewed join [Spark Branch]
> --
>
> Key: HIVE-8406
> URL: https://issues.apache.org/jira/browse/HIVE-8406
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Rui Li
>
> Research on how to handle skewed join for hive on spark. Here is original 
> hive's design doc for skewed join, 
> https://cwiki.apache.org/confluence/display/Hive/Skewed+Join+Optimization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Review Request 26481: HIVE-8411: Support partial partition spec for certain ALTER PARTITION statements

2014-10-08 Thread Jason Dere

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26481/
---

Review request for hive, Gunther Hagleitner and Thejas Nair.


Bugs: HIVE-8411
https://issues.apache.org/jira/browse/HIVE-8411


Repository: hive-git


Description
---

Allow partial partition spec for some alter table statements, such as:

alter table alter_partition_change_col1 partition (p1) change column c2 c2 
decimal(10,0);


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java d5374bc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 4e58ad8 
  ql/src/test/queries/clientpositive/alter_partition_change_col.q baabb9f 
  ql/src/test/results/clientpositive/alter_partition_change_col.q.out 7123e40 

Diff: https://reviews.apache.org/r/26481/diff/


Testing
---

updated qfile test


Thanks,

Jason Dere



[jira] [Updated] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements

2014-10-08 Thread Jason Dere (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8411:
-
Attachment: HIVE-8411.1.patch

patch v1. Supports partial partition spec for the following atler table types:

change column
add/replace columns
add/drop properties
add/drop serde properties
file format

> Support partial partition spec for certain ALTER PARTITION statements
> -
>
> Key: HIVE-8411
> URL: https://issues.apache.org/jira/browse/HIVE-8411
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-8411.1.patch
>
>
> To help address concerns [~hagleitn] had about having to update many 
> partitions 
> [here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements

2014-10-08 Thread Jason Dere (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8411:
-
Status: Patch Available  (was: Open)

> Support partial partition spec for certain ALTER PARTITION statements
> -
>
> Key: HIVE-8411
> URL: https://issues.apache.org/jira/browse/HIVE-8411
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-8411.1.patch
>
>
> To help address concerns [~hagleitn] had about having to update many 
> partitions 
> [here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HIVE-8411) Support partial partition spec for certain ALTER PARTITION statements

2014-10-08 Thread Jason Dere (JIRA)
Jason Dere created HIVE-8411:


 Summary: Support partial partition spec for certain ALTER 
PARTITION statements
 Key: HIVE-8411
 URL: https://issues.apache.org/jira/browse/HIVE-8411
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere


To help address concerns [~hagleitn] had about having to update many partitions 
[here|https://issues.apache.org/jira/browse/HIVE-7971?focusedCommentId=14131948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14131948]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-7868) AvroSerDe error handling could be improved

2014-10-08 Thread Ferdinand Xu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferdinand Xu updated HIVE-7868:
---
Attachment: HIVE-7868.2.patch

> AvroSerDe error handling could be improved
> --
>
> Key: HIVE-7868
> URL: https://issues.apache.org/jira/browse/HIVE-7868
> Project: Hive
>  Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Ferdinand Xu
> Attachments: HIVE-7868.1.patch, HIVE-7868.2.patch
>
>
> When an Avro schema is invalid, AvroSerDe returns an error message instead of 
> throwing an exception. This is described in 
> {{AvroSerdeUtils.determineSchemaOrReturnErrorSchema}}:
> {noformat}
>   /**
>* Attempt to determine the schema via the usual means, but do not throw
>* an exception if we fail.  Instead, signal failure via a special
>* schema.  This is used because Hive calls init on the serde during
>* any call, including calls to update the serde properties, meaning
>* if the serde is in a bad state, there is no way to update that state.
>*/
> {noformat}
> I believe we should find a way to provide a better experience to our users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc

2014-10-08 Thread Zhichun Wu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164627#comment-14164627
 ] 

Zhichun Wu commented on HIVE-8398:
--

attach HIVE-8398.2.patch for trunk.

> ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
> -
>
> Key: HIVE-8398
> URL: https://issues.apache.org/jira/browse/HIVE-8398
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Zhichun Wu
>Assignee: Zhichun Wu
> Attachments: HIVE-8398.2.patch, HIVE-8398.patch
>
>
> The following explain statement would fail  in hive 0.13 and trunk
> with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception:
> {code}
> create table test.t2( key string, value int);
> explain select
>sum(u.value) value
> from test.t2 u
> group by u.key
> having sum(u.value) > 30;
> {code}
> The full stack trace:
> {code}
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> I think it's due to HIVE-3107.  HIVE-3107 introduces alternate mapping for a 
> column in RowResolver.  While mapping the having clause in 
> TypeCheckProcFactory, it first maps "value" to col_1(output of groupby 
> clause) which has type of  ExprNodeColumnDesc (Before HIVE-3107, "value" is 
> not recognized).  When it comes to "u.value", it finds that "u" is a table 
> alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc.  
> Here I think we can use  the text attribute in the expr node as colAlias 
> instead.



--
This message wa

[jira] [Updated] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc

2014-10-08 Thread Zhichun Wu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhichun Wu updated HIVE-8398:
-
Status: Open  (was: Patch Available)

> ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
> -
>
> Key: HIVE-8398
> URL: https://issues.apache.org/jira/browse/HIVE-8398
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Zhichun Wu
>Assignee: Zhichun Wu
> Attachments: HIVE-8398.2.patch, HIVE-8398.patch
>
>
> The following explain statement would fail  in hive 0.13 and trunk
> with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception:
> {code}
> create table test.t2( key string, value int);
> explain select
>sum(u.value) value
> from test.t2 u
> group by u.key
> having sum(u.value) > 30;
> {code}
> The full stack trace:
> {code}
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> I think it's due to HIVE-3107.  HIVE-3107 introduces alternate mapping for a 
> column in RowResolver.  While mapping the having clause in 
> TypeCheckProcFactory, it first maps "value" to col_1(output of groupby 
> clause) which has type of  ExprNodeColumnDesc (Before HIVE-3107, "value" is 
> not recognized).  When it comes to "u.value", it finds that "u" is a table 
> alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc.  
> Here I think we can use  the text attribute in the expr node as colAlias 
> instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc

2014-10-08 Thread Zhichun Wu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhichun Wu updated HIVE-8398:
-
Status: Patch Available  (was: Open)

> ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
> -
>
> Key: HIVE-8398
> URL: https://issues.apache.org/jira/browse/HIVE-8398
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Zhichun Wu
>Assignee: Zhichun Wu
> Attachments: HIVE-8398.2.patch, HIVE-8398.patch
>
>
> The following explain statement would fail  in hive 0.13 and trunk
> with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception:
> {code}
> create table test.t2( key string, value int);
> explain select
>sum(u.value) value
> from test.t2 u
> group by u.key
> having sum(u.value) > 30;
> {code}
> The full stack trace:
> {code}
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> I think it's due to HIVE-3107.  HIVE-3107 introduces alternate mapping for a 
> column in RowResolver.  While mapping the having clause in 
> TypeCheckProcFactory, it first maps "value" to col_1(output of groupby 
> clause) which has type of  ExprNodeColumnDesc (Before HIVE-3107, "value" is 
> not recognized).  When it comes to "u.value", it finds that "u" is a table 
> alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc.  
> Here I think we can use  the text attribute in the expr node as colAlias 
> instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8398) ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc

2014-10-08 Thread Zhichun Wu (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhichun Wu updated HIVE-8398:
-
Attachment: HIVE-8398.2.patch

> ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
> -
>
> Key: HIVE-8398
> URL: https://issues.apache.org/jira/browse/HIVE-8398
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Zhichun Wu
>Assignee: Zhichun Wu
> Attachments: HIVE-8398.2.patch, HIVE-8398.patch
>
>
> The following explain statement would fail  in hive 0.13 and trunk
> with "ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc" exception:
> {code}
> create table test.t2( key string, value int);
> explain select
>sum(u.value) value
> from test.t2 u
> group by u.key
> having sum(u.value) > 30;
> {code}
> The full stack trace:
> {code}
> java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc cannot be cast to 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1067)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:184)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:9561)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9517)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:9488)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2314)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFilterPlan(SemanticAnalyzer.java:2295)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genHavingPlan(SemanticAnalyzer.java:2139)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8170)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8133)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8963)
> at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9216)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at 
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:64)
> at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:422)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> I think it's due to HIVE-3107.  HIVE-3107 introduces alternate mapping for a 
> column in RowResolver.  While mapping the having clause in 
> TypeCheckProcFactory, it first maps "value" to col_1(output of groupby 
> clause) which has type of  ExprNodeColumnDesc (Before HIVE-3107, "value" is 
> not recognized).  When it comes to "u.value", it finds that "u" is a table 
> alias but fails to cast nodeOutputs\[1\] to ExprNodeConstantDesc.  
> Here I think we can use  the text attribute in the expr node as colAlias 
> instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164623#comment-14164623
 ] 

Thejas M Nair commented on HIVE-8408:
-

Note that with V2 authorizer, it is only the additional hcat cli checks that 
get disabled. The regular checks that would happen with hive cli would still 
happen.


> hcat cli throws NPE when authorizer using new api is enabled
> 
>
> Key: HIVE-8408
> URL: https://issues.apache.org/jira/browse/HIVE-8408
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HCatalog
>Reporter: Jagruti Varia
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8408.1.patch
>
>
> Hcat cli tries to authorize actions using the authorizer configured for 
> hive-cli. When authorizer based on new authorization interface is used, it 
> throws a NPE as it checks only for the old api based authorization 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-8408:

Component/s: HCatalog
 Authorization

> hcat cli throws NPE when authorizer using new api is enabled
> 
>
> Key: HIVE-8408
> URL: https://issues.apache.org/jira/browse/HIVE-8408
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HCatalog
>Reporter: Jagruti Varia
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8408.1.patch
>
>
> Hcat cli tries to authorize actions using the authorizer configured for 
> hive-cli. When authorizer based on new authorization interface is used, it 
> throws a NPE as it checks only for the old api based authorization 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-8408:

Status: Patch Available  (was: Open)

> hcat cli throws NPE when authorizer using new api is enabled
> 
>
> Key: HIVE-8408
> URL: https://issues.apache.org/jira/browse/HIVE-8408
> Project: Hive
>  Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8408.1.patch
>
>
> Hcat cli tries to authorize actions using the authorizer configured for 
> hive-cli. When authorizer based on new authorization interface is used, it 
> throws a NPE as it checks only for the old api based authorization 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Review Request 26480: HIVE-8408 : hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas Nair

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26480/
---

Review request for hive and Sushanth Sowmyan.


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-8408


Diffs
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java
 172ff01 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatAuthUtil.java
 PRE-CREATION 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java
 0184dc0 
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/TestHCatAuthUtil.java
 PRE-CREATION 
  hcatalog/pom.xml 4b75ef5 

Diff: https://reviews.apache.org/r/26480/diff/


Testing
---

Unit tests, manual tests


Thanks,

Thejas Nair



[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-8408:

Attachment: HIVE-8408.1.patch

Hcat does an additional layer of authorization checks using the current 
authorizer configured for hive cli. It is designed keeping storage based 
authorization provider in client side in mind. 
It does not make sense for it to try and use V2 api based authorizers (storage 
based authorization in client side uses the v1 api).


> hcat cli throws NPE when authorizer using new api is enabled
> 
>
> Key: HIVE-8408
> URL: https://issues.apache.org/jira/browse/HIVE-8408
> Project: Hive
>  Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8408.1.patch
>
>
> Hcat cli tries to authorize actions using the authorizer configured for 
> hive-cli. When authorizer based on new authorization interface is used, it 
> throws a NPE as it checks only for the old api based authorization 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8364:
-
Resolution: Duplicate
Status: Resolved  (was: Patch Available)

> We're not waiting for all inputs in MapRecordProcessor on Tez
> -
>
> Key: HIVE-8364
> URL: https://issues.apache.org/jira/browse/HIVE-8364
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
> Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch
>
>
> Seems like this could be a race condition: We're blocking for some inputs to 
> become available, but the main MR input is just assumed ready...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7953) Investigate query failures (2)

2014-10-08 Thread Thomas Friedrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164603#comment-14164603
 ] 

Thomas Friedrich commented on HIVE-7953:


Thanks, Xuefu. I will create the JIRAs once I ran all the test cases of this 
JIRA.

> Investigate query failures (2)
> --
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, 
> but we should understand why they failed with an exception. This JIRA is to 
> look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external
>   0.33 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers
>   4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
> 11 sec  2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
> 0.65 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
>   4.7 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
>   2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2
>   5.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position 
> 1.5 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external
> 2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
> 11 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11   
> 5.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
>   10 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join 
> 5.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part
>  0.81 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 
>0.31 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1
> 0.26 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2  
> 0.73 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
> 8.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5  
> 0.34 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata   
> 0.93 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
>  6.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 
>2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 
>0.56 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23
>   3.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 
>4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part
> 3.2 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external
> 2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed
>  3.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_20_part_managed_location
> 3.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_19_00_part_external_location
> 6.9 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_external_table_with_space_in_location_path
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7953) Investigate query failures (2)

2014-10-08 Thread Thomas Friedrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164597#comment-14164597
 ] 

Thomas Friedrich commented on HIVE-7953:


The 4 tests
bucketsortoptimize_insert_2
bucketsortoptimize_insert_4
bucketsortoptimize_insert_7
bucketsortoptimize_insert_8
all fail with the same NPE related to SMB joins:

order object is null in SMBMapJoinOperator: 
  // fetch the first group for all small table aliases
  for (byte pos = 0; pos < order.length; pos++) {
if (pos != posBigTable) {
  fetchNextGroup(pos);
}

Daemon Thread [Executor task launch worker-3] (Suspended (exception 
NullPointerException))  
SMBMapJoinOperator.processOp(Object, int) line: 258 
FilterOperator(Operator).forward(Object, ObjectInspector) line: 799  
FilterOperator.processOp(Object, int) line: 137 
TableScanOperator(Operator).forward(Object, ObjectInspector) line: 
799   
TableScanOperator.processOp(Object, int) line: 95   
MapOperator(Operator).forward(Object, ObjectInspector) line: 799 
MapOperator.process(Writable) line: 536 
SparkMapRecordHandler.processRow(Object, Object) line: 139  

HiveMapFunctionResultList.processNextRecord(Tuple2)
 line: 47   
HiveMapFunctionResultList.processNextRecord(Object) line: 28
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108   
Wrappers$JIteratorWrapper.hasNext() line: 41 
Iterator$class.foreach(Iterator, Function1) line: 727   

Wrappers$JIteratorWrapper(AbstractIterator).foreach(Function1) line: 
1157
RDD$$anonfun$foreach$1.apply(Iterator) line: 760 
RDD$$anonfun$foreach$1.apply(Object) line: 760  
SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator) line: 
1118   
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118 
ResultTask.runTask(TaskContext) line: 61   
ResultTask(Task).run(long) line: 56 
Executor$TaskRunner.run() line: 182 
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145  
ThreadPoolExecutor$Worker.run() line: 615   
Thread.run() line: 745  

There is also a NPE in the FileSinkOperator: the FileSystem object fs is null: 
  // in recent hadoop versions, use deleteOnExit to clean tmp files.
  if (isNativeTable) {
autoDelete = fs.deleteOnExit(fsp.outPaths[0]);

Daemon Thread [Executor task launch worker-1] (Suspended (exception 
NullPointerException))  
FileSinkOperator.createBucketFiles(FileSinkOperator$FSPaths) line: 495  
FileSinkOperator.closeOp(boolean) line: 925 
FileSinkOperator(Operator).close(boolean) line: 582  
SelectOperator(Operator).close(boolean) line: 594
SMBMapJoinOperator(Operator).close(boolean) line: 594
DummyStoreOperator(Operator).close(boolean) line: 594
FilterOperator(Operator).close(boolean) line: 594
TableScanOperator(Operator).close(boolean) line: 594 
MapOperator(Operator).close(boolean) line: 594   
SparkMapRecordHandler.close() line: 175 
HiveMapFunctionResultList.closeRecordProcessor() line: 57   
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 122   
Wrappers$JIteratorWrapper.hasNext() line: 41 
Iterator$class.foreach(Iterator, Function1) line: 727   

Wrappers$JIteratorWrapper(AbstractIterator).foreach(Function1) line: 
1157
RDD$$anonfun$foreach$1.apply(Iterator) line: 760 
RDD$$anonfun$foreach$1.apply(Object) line: 760  
SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator) line: 
1118   
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118 
ResultTask.runTask(TaskContext) line: 61   
ResultTask(Task).run(long) line: 56 
Executor$TaskRunner.run() line: 182 
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145  
ThreadPoolExecutor$Worker.run() line: 615   
Thread.run() line: 745  


> Investigate query failures (2)
> --
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, 
> but we should understand why they failed with an exception. This JIRA is to 
> look into these failures and document

[jira] [Commented] (HIVE-8364) We're not waiting for all inputs in MapRecordProcessor on Tez

2014-10-08 Thread Vikram Dixit K (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164596#comment-14164596
 ] 

Vikram Dixit K commented on HIVE-8364:
--

Superseded by HIVE-8409.

> We're not waiting for all inputs in MapRecordProcessor on Tez
> -
>
> Key: HIVE-8364
> URL: https://issues.apache.org/jira/browse/HIVE-8364
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
> Attachments: HIVE-8364.1.patch, HIVE-8364.2.patch
>
>
> Seems like this could be a race condition: We're blocking for some inputs to 
> become available, but the main MR input is just assumed ready...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8409:
-
Status: Patch Available  (was: Open)

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8021) CBO: support CTAS and insert ... select

2014-10-08 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164590#comment-14164590
 ] 

Sergey Shelukhin commented on HIVE-8021:


at least 3 tests fail with CTAS on top of existing CBO failures/out file 
changes. One of them is caused by general select issue that is not uncovered by 
other q files; I haven't yet looked at others, will do so soon

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.01.patch, 
> HIVE-8021.02.patch, HIVE-8021.03.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HIVE-8410) Typo in DOAP - incorrect category URL

2014-10-08 Thread Sebb (JIRA)
Sebb created HIVE-8410:
--

 Summary: Typo in DOAP - incorrect category URL
 Key: HIVE-8410
 URL: https://issues.apache.org/jira/browse/HIVE-8410
 Project: Hive
  Issue Type: Bug
 Environment: http://svn.apache.org/repos/asf/hive/trunk/doap_Hive.rdf
Reporter: Sebb


The DOAP contains the following:

{code}
http://www.apache.org/category/database"; />
{code}

However, the URL is incorrect; it must be

{code}
http://projects.apache.org/category/database"; />
{code}

Please fix this



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8409:
-
Attachment: (was: HIVE-8409.1.patch)

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8409:
-
Attachment: HIVE-8409.1.patch

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8409:
-
Status: Open  (was: Patch Available)

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8409:
-
Attachment: HIVE-8409.1.patch

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-8409:
-
Status: Patch Available  (was: Open)

> SMB joins fail intermittently on tez
> 
>
> Key: HIVE-8409
> URL: https://issues.apache.org/jira/browse/HIVE-8409
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8409.1.patch
>
>
> Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete 
> the fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HIVE-8409) SMB joins fail intermittently on tez

2014-10-08 Thread Vikram Dixit K (JIRA)
Vikram Dixit K created HIVE-8409:


 Summary: SMB joins fail intermittently on tez
 Key: HIVE-8409
 URL: https://issues.apache.org/jira/browse/HIVE-8409
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: 0.14.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Priority: Critical
 Fix For: 0.14.0


Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the 
fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-4329) HCatalog should use getHiveRecordWriter rather than getRecordWriter

2014-10-08 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164575#comment-14164575
 ] 

Brock Noland commented on HIVE-4329:


Could you open a follow-on ticket for the parquet static partitioning issue and 
link it to HIVE-8120?

> HCatalog should use getHiveRecordWriter rather than getRecordWriter
> ---
>
> Key: HIVE-4329
> URL: https://issues.apache.org/jira/browse/HIVE-4329
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Serializers/Deserializers
>Affects Versions: 0.14.0
> Environment: discovered in Pig, but it looks like the root cause 
> impacts all non-Hive users
>Reporter: Sean Busbey
>Assignee: David Chen
> Attachments: HIVE-4329.0.patch, HIVE-4329.1.patch, HIVE-4329.2.patch, 
> HIVE-4329.3.patch, HIVE-4329.4.patch
>
>
> Attempting to write to a HCatalog defined table backed by the AvroSerde fails 
> with the following stacktrace:
> {code}
> java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be 
> cast to org.apache.hadoop.io.LongWritable
>   at 
> org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84)
>   at 
> org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253)
>   at 
> org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
>   at 
> org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242)
>   at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
>   at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
>   at 
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
>   at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> {code}
> The proximal cause of this failure is that the AvroContainerOutputFormat's 
> signature mandates a LongWritable key and HCat's FileRecordWriterContainer 
> forces a NullWritable. I'm not sure of a general fix, other than redefining 
> HiveOutputFormat to mandate a WritableComparable.
> It looks like accepting WritableComparable is what's done in the other Hive 
> OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also 
> be changed, since it's ignoring the key. That way fixing things so 
> FileRecordWriterContainer can always use NullWritable could get spun into a 
> different issue?
> The underlying cause for failure to write to AvroSerde tables is that 
> AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so 
> fixing the above will just push the failure into the placeholder RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8292) Reading from partitioned bucketed tables has high overhead in MapOperator.cleanUpInputFileChangedOp

2014-10-08 Thread Gopal V (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-8292:
--
Assignee: Gopal V  (was: Vikram Dixit K)
  Status: Patch Available  (was: Open)

> Reading from partitioned bucketed tables has high overhead in 
> MapOperator.cleanUpInputFileChangedOp
> ---
>
> Key: HIVE-8292
> URL: https://issues.apache.org/jira/browse/HIVE-8292
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: cn105
>Reporter: Mostafa Mokhtar
>Assignee: Gopal V
> Fix For: 0.14.0
>
> Attachments: 2014_09_29_14_46_04.jfr, HIVE-8292.1.patch, 
> HIVE-8292.2.patch
>
>
> Reading from bucketed partitioned tables has significantly higher overhead 
> compared to non-bucketed non-partitioned files.
> 50% of the profile is spent in MapOperator.cleanUpInputFileChangedOp
> 5% the CPU in 
> {code}
>  Path onepath = normalizePath(onefile);
> {code}
> And 
> 45% the CPU in 
> {code}
>  onepath.toUri().relativize(fpath.toUri()).equals(fpath.toUri());
> {code}
> From the profiler 
> {code}
> Stack Trace   Sample CountPercentage(%)
> hive.ql.exec.tez.MapRecordSource.processRow(Object)   5,327   62.348
>hive.ql.exec.vector.VectorMapOperator.process(Writable)5,326   62.336
>   hive.ql.exec.Operator.cleanUpInputFileChanged() 4,851   56.777
>  hive.ql.exec.MapOperator.cleanUpInputFileChangedOp() 4,849   56.753
>  java.net.URI.relativize(URI) 3,903   45.681
> java.net.URI.relativize(URI, URI) 3,903   
> 45.681
>java.net.URI.normalize(String) 2,169   
> 25.386
>java.net.URI.equal(String, String) 
> 526 6.156
>java.net.URI.equalIgnoringCase(String, 
> String) 1   0.012
>java.lang.String.substring(int)
> 1   0.012
> hive.ql.exec.MapOperator.normalizePath(String)506 5.922
> org.apache.commons.logging.impl.Log4JLogger.info(Object)  32  
> 0.375
>  java.net.URI.equals(Object)  12  0.14
>  java.util.HashMap$KeySet.iterator()  5   
> 0.059
>  java.util.HashMap.get(Object)4   
> 0.047
>  java.util.LinkedHashMap.get(Object)  3   
> 0.035
>  hive.ql.exec.Operator.cleanUpInputFileChanged()  1   0.012
>   hive.ql.exec.Operator.forward(Object, ObjectInspector)  473 5.536
>   hive.ql.exec.mr.ExecMapperContext.inputFileChanged()1   0.012
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (HIVE-8120) Umbrella JIRA tracking Parquet improvements

2014-10-08 Thread Brock Noland (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland reassigned HIVE-8120:
--

Assignee: Brock Noland

> Umbrella JIRA tracking Parquet improvements
> ---
>
> Key: HIVE-8120
> URL: https://issues.apache.org/jira/browse/HIVE-8120
> Project: Hive
>  Issue Type: Improvement
>Reporter: Brock Noland
>Assignee: Brock Noland
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-6914) parquet-hive cannot write nested map (map value is map)

2014-10-08 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164563#comment-14164563
 ] 

Brock Noland commented on HIVE-6914:


Linking to HIVE-8120.

FYI [~spena] who is looking into some parquet hive schema stuff as well.

> parquet-hive cannot write nested map (map value is map)
> ---
>
> Key: HIVE-6914
> URL: https://issues.apache.org/jira/browse/HIVE-6914
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Tongjie Chen
>  Labels: parquet, serialization
> Attachments: HIVE-6914.1.patch
>
>
> // table schema (identical for both plain text version and parquet version)
> desc hive> desc text_mmap;
> m map>
> // sample nested map entry
> {"level1":{"level2_key1":"value1","level2_key2":"value2"}}
> The following query will fail, 
> insert overwrite table parquet_mmap select * from text_mmap;
> Caused by: parquet.io.ParquetEncodingException: This should be an 
> ArrayWritable or MapWritable: 
> org.apache.hadoop.hive.ql.io.parquet.writable.BinaryWritable@f2f8106
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:85)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeArray(DataWritableWriter.java:118)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:80)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.writeData(DataWritableWriter.java:82)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:55)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at 
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:115)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:81)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:37)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:77)
> at 
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:90)
> at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:622)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
> ... 9 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8395) CBO: enable by default

2014-10-08 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164561#comment-14164561
 ] 

Hive QA commented on HIVE-8395:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12673670/HIVE-8395.01.patch

{color:red}ERROR:{color} -1 due to 356 failed/errored test(s), 6542 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver_accumulo_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ansi_sql_arithmetic
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_explain
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_binarysortable_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_count
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_genericudaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_distinct_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_distinct_samekey
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_position
org.apache.hadoop.hive.cli.Tes

[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large

2014-10-08 Thread Gunther Hagleitner (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164553#comment-14164553
 ] 

Gunther Hagleitner commented on HIVE-8341:
--

I use this script:

{noformat}
#!/usr/bin/python

import sys
import random

for line in sys.stdin:
print random.random()
{noformat}

and this query:

{noformat}
add FILE /home/gunther/rand.py;
select transform () using 'rand.py' as (foo float) from ;
{noformat}

Does that help?



> Transaction information in config file can grow excessively large
> -
>
> Key: HIVE-8341
> URL: https://issues.apache.org/jira/browse/HIVE-8341
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8341.patch
>
>
> In our testing we have seen cases where the transaction list grows very 
> large.  We need a more efficient way of communicating the list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity

2014-10-08 Thread Prasad Mujumdar (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164544#comment-14164544
 ] 

Prasad Mujumdar commented on HIVE-7932:
---

[~vikram.dixit] Requesting backport to 0.14. It's a followup to HIVE-7730 which 
is already in 0.14. Thanks!

> It may cause NP exception when add accessed columns to ReadEntity
> -
>
> Key: HIVE-7932
> URL: https://issues.apache.org/jira/browse/HIVE-7932
> Project: Hive
>  Issue Type: Bug
>Reporter: Xiaomeng Huang
>Assignee: Xiaomeng Huang
> Fix For: 0.15.0
>
> Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
>
>
> {code}
> case TABLE:
>entity.getAccessedColumns().addAll(
>   tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
> {code}
> if  tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null, 
> addAll(null) will throw null pointer exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group

2014-10-08 Thread Prasad Mujumdar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-8083:
--
Release Note: The user name in the grant and revoke statements may be 
optionally surrounded by backtick characters (`) irrespective of the 
hive.support.quoted.identifiers setting.

> Authorization DDLs should not enforce hive identifier syntax for user or group
> --
>
> Key: HIVE-8083
> URL: https://issues.apache.org/jira/browse/HIVE-8083
> Project: Hive
>  Issue Type: Bug
>  Components: SQL, SQLStandardAuthorization
>Affects Versions: 0.13.0, 0.13.1
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch
>
>
> The compiler expects principals (user, group and role) as hive identifiers 
> for authorization DDLs. The user and group are entities that belong to 
> external namespace and we can't expect those to follow hive identifier syntax 
> rules. For example, a userid or group can contain '-' which is not allowed by 
> compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8083) Authorization DDLs should not enforce hive identifier syntax for user or group

2014-10-08 Thread Prasad Mujumdar (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164534#comment-14164534
 ] 

Prasad Mujumdar commented on HIVE-8083:
---

[~leftylev] Thanks for pointing that out. Just updated the wiki.

> Authorization DDLs should not enforce hive identifier syntax for user or group
> --
>
> Key: HIVE-8083
> URL: https://issues.apache.org/jira/browse/HIVE-8083
> Project: Hive
>  Issue Type: Bug
>  Components: SQL, SQLStandardAuthorization
>Affects Versions: 0.13.0, 0.13.1
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-8083.1.patch, HIVE-8083.2.patch, HIVE-8083.3.patch
>
>
> The compiler expects principals (user, group and role) as hive identifiers 
> for authorization DDLs. The user and group are entities that belong to 
> external namespace and we can't expect those to follow hive identifier syntax 
> rules. For example, a userid or group can contain '-' which is not allowed by 
> compiler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity

2014-10-08 Thread Prasad Mujumdar (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasad Mujumdar updated HIVE-7932:
--
   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks [~Huang Xiaomeng]!

> It may cause NP exception when add accessed columns to ReadEntity
> -
>
> Key: HIVE-7932
> URL: https://issues.apache.org/jira/browse/HIVE-7932
> Project: Hive
>  Issue Type: Bug
>Reporter: Xiaomeng Huang
>Assignee: Xiaomeng Huang
> Fix For: 0.15.0
>
> Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
>
>
> {code}
> case TABLE:
>entity.getAccessedColumns().addAll(
>   tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
> {code}
> if  tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null, 
> addAll(null) will throw null pointer exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8374) schematool fails on Postgres versions < 9.2

2014-10-08 Thread Mohit Sabharwal (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164500#comment-14164500
 ] 

Mohit Sabharwal commented on HIVE-8374:
---

Instead of having the user supply an option dependent on Postgres version, 
changed to the patch to automatically run a "pre-upgrade" script that creates 
the language, but any error is ignored.

Kept the dbOpts option as that is useful when we add any db specific options.


> schematool fails on Postgres versions < 9.2
> ---
>
> Key: HIVE-8374
> URL: https://issues.apache.org/jira/browse/HIVE-8374
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-8374.1.patch, HIVE-8374.patch
>
>
> The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql',
> which is available by default only for Postgres 9.2+.
> For older Postgres versions, the language must be explicitly created,
> otherwise schematool fails with the error:
> {code}
> Error: ERROR: language "plpgsql" does not exist
>   Hint: Use CREATE LANGUAGE to load the language into the database. 
> (state=42704,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8374) schematool fails on Postgres versions < 9.2

2014-10-08 Thread Mohit Sabharwal (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mohit Sabharwal updated HIVE-8374:
--
Attachment: HIVE-8374.1.patch

> schematool fails on Postgres versions < 9.2
> ---
>
> Key: HIVE-8374
> URL: https://issues.apache.org/jira/browse/HIVE-8374
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-8374.1.patch, HIVE-8374.patch
>
>
> The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql',
> which is available by default only for Postgres 9.2+.
> For older Postgres versions, the language must be explicitly created,
> otherwise schematool fails with the error:
> {code}
> Error: ERROR: language "plpgsql" does not exist
>   Hint: Use CREATE LANGUAGE to load the language into the database. 
> (state=42704,code=0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 26461: [CBO] Handle filters with non-boolean return type

2014-10-08 Thread John Pullokkaran

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26461/#review55923
---

Ship it!


Ship It!

- John Pullokkaran


On Oct. 8, 2014, 10:23 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26461/
> ---
> 
> (Updated Oct. 8, 2014, 10:23 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-8407
> https://issues.apache.org/jira/browse/HIVE-8407
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> [CBO] Handle filters with non-boolean return type
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 19110ce 
> 
> Diff: https://reviews.apache.org/r/26461/diff/
> 
> 
> Testing
> ---
> 
> annotate_stats_filter.q
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>



Re: Review Request 26403: HIVE-8374: schematool fails on Postgres versions < 9.2

2014-10-08 Thread Mohit Sabharwal

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26403/
---

(Updated Oct. 9, 2014, 12:51 a.m.)


Review request for hive.


Changes
---

Instead of having the user supply an option dependent on Postgres
version, changed to the patch to automatically run a "pre-upgrade"
script that creates the language, but any error is ignored.

Kept the dbOpts option as that is useful when we add any 
db specific options.


Bugs: HIVE-8374
https://issues.apache.org/jira/browse/HIVE-8374


Repository: hive-git


Description (updated)
---

HIVE-8374: schematool fails on Postgres versions < 9.2

The upgrade script for HIVE-5700 creates an UDF with language 'plpgsql',
which is available by default only for Postgres 9.2+.

For older Postgres versions, the language must be explicitly created,
otherwise schematool fails.

This patch allows for running "pre-upgrade" scripts before running a
upgrade script. The errors from pre-upgrade scripts are ignored.
This enables us to run statements like "create language" as a
pre-upgrade step and ignore failures.

This patch also adds a command line option to schematool to specify
db options. This will be useful for adding future options that
can conditionally execute certain statements.


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java 
a21fa652e9c1aa3fd763afa4136f6fcb7c7b7459 
  beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
c376687cb47332323912e4c6dbe713b7b37ae834 
  itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java 
749fb9e86b4f74f768da356cf82f621fdef399cd 
  metastore/scripts/upgrade/postgres/015-HIVE-5700.postgres.sql 
aedb2b5dbf9b513cb7c124f59e05ea9831455a72 
  
metastore/scripts/upgrade/postgres/pre-0-upgrade-0.12.0-to-0.13.0.postgres.sql 
PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java 
af56ef6a7c6e27ead312a68d30d94802095f2c60 

Diff: https://reviews.apache.org/r/26403/diff/


Testing
---

Added unit test


Thanks,

Mohit Sabharwal



[jira] [Commented] (HIVE-7953) Investigate query failures (2)

2014-10-08 Thread Xuefu Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164475#comment-14164475
 ] 

Xuefu Zhang commented on HIVE-7953:
---

Hi [~tfriedr], thank you very much for the nice analysis. After you have gone 
thru all the tests in the set, please create individual JIRAs to track each 
issue that you find. Thanks.

> Investigate query failures (2)
> --
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, 
> but we should understand why they failed with an exception. This JIRA is to 
> look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external
>   0.33 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers
>   4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
> 11 sec  2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
> 0.65 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
>   4.7 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
>   2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2
>   5.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position 
> 1.5 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external
> 2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
> 11 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11   
> 5.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
>   10 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join 
> 5.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part
>  0.81 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 
>0.31 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1
> 0.26 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2  
> 0.73 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
> 8.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5  
> 0.34 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata   
> 0.93 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
>  6.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 
>2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 
>0.56 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23
>   3.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 
>4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part
> 3.2 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external
> 2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed
>  3.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_20_part_managed_location
> 3.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_19_00_part_external_location
> 6.9 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_external_table_with_space_in_location_path
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large

2014-10-08 Thread Alan Gates (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164472#comment-14164472
 ] 

Alan Gates commented on HIVE-8341:
--

Do you have a simple query with a transform in it that shows the issue with the 
process builder?

> Transaction information in config file can grow excessively large
> -
>
> Key: HIVE-8341
> URL: https://issues.apache.org/jira/browse/HIVE-8341
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8341.patch
>
>
> In our testing we have seen cases where the transaction list grows very 
> large.  We need a more efficient way of communicating the list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large

2014-10-08 Thread Gunther Hagleitner (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164458#comment-14164458
 ] 

Gunther Hagleitner commented on HIVE-8341:
--

I'm not sure what values ProcEnv supports, but maybe base64 would be better?

> Transaction information in config file can grow excessively large
> -
>
> Key: HIVE-8341
> URL: https://issues.apache.org/jira/browse/HIVE-8341
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8341.patch
>
>
> In our testing we have seen cases where the transaction list grows very 
> large.  We need a more efficient way of communicating the list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8341) Transaction information in config file can grow excessively large

2014-10-08 Thread Gunther Hagleitner (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164454#comment-14164454
 ] 

Gunther Hagleitner commented on HIVE-8341:
--

This made it worse, the string encoding you're doing doesn't work with the 
process builder:

{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid environment variable 
value: 
at 
java.lang.ProcessEnvironment.validateValue(ProcessEnvironment.java:120)
at java.lang.ProcessEnvironment.access$400(ProcessEnvironment.java:61)
at 
java.lang.ProcessEnvironment$Value.valueOf(ProcessEnvironment.java:202)
at 
java.lang.ProcessEnvironment$StringEnvironment.put(ProcessEnvironment.java:241)
at 
java.lang.ProcessEnvironment$StringEnvironment.put(ProcessEnvironment.java:220)
at 
org.apache.hadoop.hive.ql.exec.ScriptOperator.addJobConfToEnvironment(ScriptOperator.java:156)
at 
org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:317)
{noformat}

> Transaction information in config file can grow excessively large
> -
>
> Key: HIVE-8341
> URL: https://issues.apache.org/jira/browse/HIVE-8341
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Attachments: HIVE-8341.patch
>
>
> In our testing we have seen cases where the transaction list grows very 
> large.  We need a more efficient way of communicating the list.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8404) ColumnPruner doesnt prune columns from limit operator

2014-10-08 Thread Ashutosh Chauhan (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8404:
---
   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

Failures are not related to patch. Also, ran those tests and cant repro them on 
top of patch.
Committed to trunk. 

> ColumnPruner doesnt prune columns from limit operator
> -
>
> Key: HIVE-8404
> URL: https://issues.apache.org/jira/browse/HIVE-8404
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 0.12.0, 0.13.0, 0.13.1
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.15.0
>
> Attachments: HIVE-8404.1.patch, HIVE-8404.patch
>
>
> Discovered while working on HIVE-8358 input26.q failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-7880) Support subquery [Spark Branch]

2014-10-08 Thread Xuefu Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-7880:
--
   Resolution: Fixed
Fix Version/s: spark-branch
   Status: Resolved  (was: Patch Available)

Patch committed to Spark branch.

> Support subquery [Spark Branch]
> ---
>
> Key: HIVE-7880
> URL: https://issues.apache.org/jira/browse/HIVE-7880
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Chengxiang Li
>Assignee: Xuefu Zhang
>  Labels: Spark-M2
> Fix For: spark-branch
>
> Attachments: HIVE-7880.1-spark.patch
>
>
> While try to enable SubQuery qtests, I found that SubQuery cases return null 
> value currently, we should enable subquery for Hive on Spark. We should 
> enable subquery_exists.q and subquery_in.q in this task as Tez does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8386) HCAT api call is case sensitive on fields in struct column

2014-10-08 Thread Jason Dere (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8386:
-
Attachment: HIVE-8386.2.patch

Re-attaching patch for pre-commit tests

> HCAT api call is case sensitive on fields in struct column
> --
>
> Key: HIVE-8386
> URL: https://issues.apache.org/jira/browse/HIVE-8386
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Types
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-8386.1.patch, HIVE-8386.2.patch
>
>
> Falcon using hcat api to verify the target table schema and getting the error:
> {noformat}
> 2014-10-07 00:30:23,255 ERROR - 
> [1972803970@qtp-1214921164-3:gfoetl:POST//entities/submitAndSchedule/feed 
> a0c221e3-efa8-4235-a403-b1047f23ec05] ~ Failure reason (FalconWebException:40)
> java.lang.RuntimeException: cannot find field givenName(lowercase form: 
> givenname) in [givenName, surname, middleName, gender, age, 
> isGivenNameLowerCase, isGivenNameUpperCase, isPrimary, isSurnameLowerCase, 
> isSurnameUpperCase]
>   at 
> org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo.getStructFieldTypeInfo(StructTypeInfo.java:109)
>   at 
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.constructHCatSchema(HCatSchemaUtils.java:154)
>   at 
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatSchema(HCatSchemaUtils.java:165)
>   at 
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:127)
>   at 
> org.apache.hive.hcatalog.data.schema.HCatSchemaUtils.getHCatFieldSchema(HCatSchemaUtils.java:115)
>   at org.apache.hive.hcatalog.api.HCatTable.(HCatTable.java:59)
>   at 
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.getTable(HCatClientHMSImpl.java:157)
>   at 
> org.apache.falcon.catalog.HiveCatalogService.tableExists(HiveCatalogService.java:143)
>   at 
> org.apache.falcon.entity.parser.FeedEntityParser.validateStorageExists(FeedEntityParser.java:367)
>   at 
> org.apache.falcon.entity.parser.FeedEntityParser.validateFeedStorage(FeedEntityParser.java:309)
>   at 
> org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:79)
>   at 
> org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:54)
>   at 
> org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:364)
>   at 
> org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:331)
>   at 
> org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:153)
>   at 
> org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:44)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:48)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$1.doExecute(SchedulableEntityManagerProxy.java:118)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:410)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody0(SchedulableEntityManagerProxy.java:120)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure1.run(SchedulableEntityManagerProxy.java:1)
>   at 
> org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>   at 
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:107)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody12(SchedulableEntityManagerProxy.java:341)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure13.run(SchedulableEntityManagerProxy.java:1)
>   at 
> org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>   at 
> org.apache.falcon.aspect.AbstractFalconAspect.logAround(AbstractFalconAspect.java:50)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submitAndSchedule_aroundBody16(SchedulableEntityManagerProxy.java:341)
>   at 
> org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure17.run(SchedulableEntityManagerProxy.java:1)
>   at 
> org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>   at 
> org.apac

[jira] [Commented] (HIVE-8404) ColumnPruner doesnt prune columns from limit operator

2014-10-08 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164421#comment-14164421
 ] 

Hive QA commented on HIVE-8404:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12673694/HIVE-8404.1.patch

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 6528 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnection
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConnectionSchemaAPIs
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testNewConnectionConfiguration
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testRootScratchDir
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testSessionScratchDirs
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testURIDatabaseName
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1181/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1181/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1181/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12673694

> ColumnPruner doesnt prune columns from limit operator
> -
>
> Key: HIVE-8404
> URL: https://issues.apache.org/jira/browse/HIVE-8404
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Affects Versions: 0.12.0, 0.13.0, 0.13.1
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8404.1.patch, HIVE-8404.patch
>
>
> Discovered while working on HIVE-8358 input26.q failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7880) Support subquery [Spark Branch]

2014-10-08 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164411#comment-14164411
 ] 

Hive QA commented on HIVE-7880:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12673728/HIVE-7880.1-spark.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6755 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parallel
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/197/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/197/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-197/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12673728

> Support subquery [Spark Branch]
> ---
>
> Key: HIVE-7880
> URL: https://issues.apache.org/jira/browse/HIVE-7880
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Chengxiang Li
>Assignee: Xuefu Zhang
>  Labels: Spark-M2
> Attachments: HIVE-7880.1-spark.patch
>
>
> While try to enable SubQuery qtests, I found that SubQuery cases return null 
> value currently, we should enable subquery for Hive on Spark. We should 
> enable subquery_exists.q and subquery_in.q in this task as Tez does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8363) AccumuloStorageHandler compile failure hadoop-1

2014-10-08 Thread Szehon Ho (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-8363:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to 0.14 branch, thanks guys.

> AccumuloStorageHandler compile failure hadoop-1
> ---
>
> Key: HIVE-8363
> URL: https://issues.apache.org/jira/browse/HIVE-8363
> Project: Hive
>  Issue Type: Bug
>  Components: StorageHandler
>Reporter: Szehon Ho
>Assignee: Josh Elser
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8363.1.patch
>
>
> There's an error about AccumuloStorageHandler compiling on hadoop-1.  It 
> seems the signature of split() is not the same.  Looks like we can should use 
> another utils to fix this.
> {code}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) 
> on project hive-accumulo-handler: Compilation failure
> [ERROR] 
> /data/hive-ptest/working/apache-svn-trunk-source/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/columns/ColumnMapper.java:[57,52]
>  no suitable method found for split(java.lang.String,char)
> [ERROR] method 
> org.apache.hadoop.util.StringUtils.split(java.lang.String,char,char) is not 
> applicable
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7953) Investigate query failures (2)

2014-10-08 Thread Thomas Friedrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164391#comment-14164391
 ] 

Thomas Friedrich commented on HIVE-7953:


More details for auto_sortmerge_join_2 and auto_sortmerge_join_3:
They fail with NPE in SMBJoinDesc: filterMap is null:
filterMap = compactFilter(filterMap);
Map result = new LinkedHashMap();
for (int i = 0 ; i < filterMap.length; i++) {
  if (filterMap[i] == null) {
continue;
  }

Thread [main] (Suspended (exception NullPointerException))  
SMBJoinDesc(JoinDesc).toCompactString(int[][]) line: 482
SMBJoinDesc(JoinDesc).getFilterMapString() line: 473
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not 
available [native method]  
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 57  
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43  
Method.invoke(Object, Object...) line: 606  
ExplainTask.outputPlan(Serializable, PrintStream, boolean, boolean, 
int, String) line: 622  


> Investigate query failures (2)
> --
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, 
> but we should understand why they failed with an exception. This JIRA is to 
> look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external
>   0.33 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers
>   4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
> 11 sec  2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
> 0.65 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
>   4.7 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
>   2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2
>   5.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position 
> 1.5 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external
> 2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
> 11 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11   
> 5.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
>   10 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join 
> 5.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part
>  0.81 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 
>0.31 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1
> 0.26 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2  
> 0.73 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
> 8.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5  
> 0.34 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata   
> 0.93 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
>  6.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 
>2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 
>0.56 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23
>   3.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 
>4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part
> 3.2 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_16_part_external
> 2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_17_part_managed
>  3.4 sec 2
>  

[jira] [Updated] (HIVE-6669) sourcing txn-script from schema script results in failure for mysql & oracle

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6669:
-
Attachment: HIVE-6669.2.patch

A new version of the patch that quotes postgres table and field names.

> sourcing txn-script from schema script results in failure for mysql & oracle
> 
>
> Key: HIVE-6669
> URL: https://issues.apache.org/jira/browse/HIVE-6669
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Prasad Mujumdar
>Assignee: Alan Gates
>Priority: Blocker
> Attachments: HIVE-6669.2.patch, HIVE-6669.patch
>
>
> This issues is addressed in 0.13 by in-lining the the transaction schema 
> statements in the schema initialization script (HIVE-6559)
> The 0.14 schema initialization is not fixed. This is the followup ticket for 
> to address the problem in 0.14. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8344) Hive on Tez sets mapreduce.framework.name to yarn-tez

2014-10-08 Thread Gunther Hagleitner (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8344:
-
   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk and hive .14

> Hive on Tez sets mapreduce.framework.name to yarn-tez
> -
>
> Key: HIVE-8344
> URL: https://issues.apache.org/jira/browse/HIVE-8344
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8344.1.patch, HIVE-8344.2.patch, HIVE-8344.3.patch
>
>
> This was done to run MR jobs when in Tez mode (emulate MR on Tez). However, 
> we don't switch back when the user specifies MR as exec engine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8258:
-
Attachment: HIVE-8258.5.patch

Rebased patch.

> Compactor cleaners can be starved on a busy table or partition.
> ---
>
> Key: HIVE-8258
> URL: https://issues.apache.org/jira/browse/HIVE-8258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.13.1
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, 
> HIVE-8258.5.patch, HIVE-8258.patch
>
>
> Currently the cleaning thread in the compactor does not run on a table or 
> partition while any locks are held on this partition.  This leaves it open to 
> starvation in the case of a busy table or partition.  It only needs to wait 
> until all locks on the table/partition at the time of the compaction have 
> expired.  Any jobs initiated after that (and thus any locks obtained) will be 
> for the new versions of the files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8258:
-
Status: Patch Available  (was: Open)

> Compactor cleaners can be starved on a busy table or partition.
> ---
>
> Key: HIVE-8258
> URL: https://issues.apache.org/jira/browse/HIVE-8258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.13.1
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, 
> HIVE-8258.5.patch, HIVE-8258.patch
>
>
> Currently the cleaning thread in the compactor does not run on a table or 
> partition while any locks are held on this partition.  This leaves it open to 
> starvation in the case of a busy table or partition.  It only needs to wait 
> until all locks on the table/partition at the time of the compaction have 
> expired.  Any jobs initiated after that (and thus any locks obtained) will be 
> for the new versions of the files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8258) Compactor cleaners can be starved on a busy table or partition.

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8258:
-
Status: Open  (was: Patch Available)

> Compactor cleaners can be starved on a busy table or partition.
> ---
>
> Key: HIVE-8258
> URL: https://issues.apache.org/jira/browse/HIVE-8258
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.13.1
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8258.2.patch, HIVE-8258.3.patch, HIVE-8258.4.patch, 
> HIVE-8258.patch
>
>
> Currently the cleaning thread in the compactor does not run on a table or 
> partition while any locks are held on this partition.  This leaves it open to 
> starvation in the case of a busy table or partition.  It only needs to wait 
> until all locks on the table/partition at the time of the compaction have 
> expired.  Any jobs initiated after that (and thus any locks obtained) will be 
> for the new versions of the files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8367) delete writes records in wrong order in some cases

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8367:
-
Attachment: HIVE-8367.2.patch

Rebased version of the patch.

> delete writes records in wrong order in some cases
> --
>
> Key: HIVE-8367
> URL: https://issues.apache.org/jira/browse/HIVE-8367
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8367.2.patch, HIVE-8367.patch
>
>
> I have found one query with 10k records where you do:
> create table
> insert into table -- 10k records
> delete from table -- just some records
> The records in the delete delta are not ordered properly by rowid.
> I assume this applies to updates as well, but I haven't tested it yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)
Thejas M Nair created HIVE-8408:
---

 Summary: hcat cli throws NPE when authorizer using new api is 
enabled
 Key: HIVE-8408
 URL: https://issues.apache.org/jira/browse/HIVE-8408
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Priority: Critical
 Fix For: 0.14.0


Hcat cli tries to authorize actions using the authorizer configured for 
hive-cli. When authorizer based on new authorization interface is used, it 
throws a NPE as it checks only for the old api based authorization 
implementation.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164362#comment-14164362
 ] 

Thejas M Nair commented on HIVE-8408:
-

The stack trace - 
{noformat}
ERROR [main]: ql.Driver (SessionState.java:printError(825)) - FAILED: 
SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: 
java.lang.NullPointerException
org.apache.hadoop.hive.ql.parse.SemanticException: 
org.apache.hadoop.hive.ql.parse.SemanticException: 
java.lang.NullPointerException
at 
org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzer.postAnalyze(HCatSemanticAnalyzer.java:266)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:418)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1129)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
at org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)
at org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: 
java.lang.NullPointerException
at 
org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzerBase.authorizeDDL(HCatSemanticAnalyzerBase.java:114)
at 
org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzer.postAnalyze(HCatSemanticAnalyzer.java:261)
... 16 more
Caused by: java.lang.NullPointerException
at 
org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzerBase.authorize(HCatSemanticAnalyzerBase.java:139)
at 
org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzer.authorizeDDLWork(HCatSemanticAnalyzer.java:324)
at 
org.apache.hive.hcatalog.cli.SemanticAnalysis.HCatSemanticAnalyzerBase.authorizeDDL(HCatSemanticAnalyzerBase.java:105)
... 17 more
{noformat}

> hcat cli throws NPE when authorizer using new api is enabled
> 
>
> Key: HIVE-8408
> URL: https://issues.apache.org/jira/browse/HIVE-8408
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
>
> Hcat cli tries to authorize actions using the authorizer configured for 
> hive-cli. When authorizer based on new authorization interface is used, it 
> throws a NPE as it checks only for the old api based authorization 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8408) hcat cli throws NPE when authorizer using new api is enabled

2014-10-08 Thread Thejas M Nair (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-8408:

Reporter: Jagruti Varia  (was: Thejas M Nair)

> hcat cli throws NPE when authorizer using new api is enabled
> 
>
> Key: HIVE-8408
> URL: https://issues.apache.org/jira/browse/HIVE-8408
> Project: Hive
>  Issue Type: Bug
>Reporter: Jagruti Varia
>Assignee: Thejas M Nair
>Priority: Critical
> Fix For: 0.14.0
>
>
> Hcat cli tries to authorize actions using the authorizer configured for 
> hive-cli. When authorizer based on new authorization interface is used, it 
> throws a NPE as it checks only for the old api based authorization 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8367) delete writes records in wrong order in some cases

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8367:
-
Status: Patch Available  (was: Open)

> delete writes records in wrong order in some cases
> --
>
> Key: HIVE-8367
> URL: https://issues.apache.org/jira/browse/HIVE-8367
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8367.2.patch, HIVE-8367.patch
>
>
> I have found one query with 10k records where you do:
> create table
> insert into table -- 10k records
> delete from table -- just some records
> The records in the delete delta are not ordered properly by rowid.
> I assume this applies to updates as well, but I haven't tested it yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8367) delete writes records in wrong order in some cases

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8367:
-
Status: Open  (was: Patch Available)

> delete writes records in wrong order in some cases
> --
>
> Key: HIVE-8367
> URL: https://issues.apache.org/jira/browse/HIVE-8367
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8367.patch
>
>
> I have found one query with 10k records where you do:
> create table
> insert into table -- 10k records
> delete from table -- just some records
> The records in the delete delta are not ordered properly by rowid.
> I assume this applies to updates as well, but I haven't tested it yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8368:
-
Status: Patch Available  (was: Open)

> compactor is improperly writing delete records in base file
> ---
>
> Key: HIVE-8368
> URL: https://issues.apache.org/jira/browse/HIVE-8368
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8368.2.patch, HIVE-8368.patch
>
>
> When the compactor reads records from the base and deltas, it is not properly 
> dropping delete records.  This leads to oversized base files, and possibly to 
> wrong query results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8368:
-
Attachment: HIVE-8368.2.patch

Rebased version of the patch.

> compactor is improperly writing delete records in base file
> ---
>
> Key: HIVE-8368
> URL: https://issues.apache.org/jira/browse/HIVE-8368
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8368.2.patch, HIVE-8368.patch
>
>
> When the compactor reads records from the base and deltas, it is not properly 
> dropping delete records.  This leads to oversized base files, and possibly to 
> wrong query results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8368:
-
Status: Open  (was: Patch Available)

> compactor is improperly writing delete records in base file
> ---
>
> Key: HIVE-8368
> URL: https://issues.apache.org/jira/browse/HIVE-8368
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8368.patch
>
>
> When the compactor reads records from the base and deltas, it is not properly 
> dropping delete records.  This leads to oversized base files, and possibly to 
> wrong query results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8402) Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8402:
-
Status: Patch Available  (was: Open)

> Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions
> -
>
> Key: HIVE-8402
> URL: https://issues.apache.org/jira/browse/HIVE-8402
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8402.patch
>
>
> ORC is in some instances pushing SARGs into delta files.  This is wrong 
> behavior in general as it may result in failing to pull the most recent 
> version of a row.  When the SARG is applied to a row that is deleted it 
> causes an ArrayOutOfBoundsException because there is no data in the row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8402) Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions

2014-10-08 Thread Alan Gates (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8402:
-
Attachment: HIVE-8402.patch

A patch to change orc to not push sargs into the deltas.

And to answer my earlier unknown,  this did only happen when a base was also 
present.  When there was no base file the sarg was not being written into the 
options passed to OrcRawRecordMerge (see OrcInputFormat.getReader, around line 
1121). 

> Orc pushing SARGs into delta files causing ArrayOutOfBoundsExceptions
> -
>
> Key: HIVE-8402
> URL: https://issues.apache.org/jira/browse/HIVE-8402
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8402.patch
>
>
> ORC is in some instances pushing SARGs into delta files.  This is wrong 
> behavior in general as it may result in failing to pull the most recent 
> version of a row.  When the SARG is applied to a row that is deleted it 
> causes an ArrayOutOfBoundsException because there is no data in the row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8393) Handle SIGINT on Tez

2014-10-08 Thread Gunther Hagleitner (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8393:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk and hive .14

> Handle SIGINT on Tez
> 
>
> Key: HIVE-8393
> URL: https://issues.apache.org/jira/browse/HIVE-8393
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-8393.1.patch, HIVE-8393.2.patch
>
>
> The signal handler in hive currently only interrupts MR jobs. The same logic 
> can be used to interrupt Tez dags as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8393) Handle SIGINT on Tez

2014-10-08 Thread Gunther Hagleitner (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8393:
-
Fix Version/s: 0.14.0

> Handle SIGINT on Tez
> 
>
> Key: HIVE-8393
> URL: https://issues.apache.org/jira/browse/HIVE-8393
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: 0.14.0
>
> Attachments: HIVE-8393.1.patch, HIVE-8393.2.patch
>
>
> The signal handler in hive currently only interrupts MR jobs. The same logic 
> can be used to interrupt Tez dags as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8367) delete writes records in wrong order in some cases

2014-10-08 Thread Alan Gates (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164332#comment-14164332
 ] 

Alan Gates commented on HIVE-8367:
--

bq. What was the original query where the issue showed up?
{code}
create table concur_orc_tab(name varchar(50), age int, gpa decimal(3, 2)) 
clustered by (age) into 2 buckets stored as orc TBLPROPERTIES 
('transactional'='true');
insert into table concur_orc_tab select * from texttab; -- loads 10k records 
into the table
delete from concur_orc_tab where age >= 20 and age < 30;
{code}
This resulted in only some rows being deleted (~300 of the 1700 that should 
have been deleted)

What precisely was the problem and how does the RS deduplication change help?
The problem was that because the code was turning off the RS deduplication it 
was getting a plan with two MR jobs.  The sort by ROW__ID was done in job one, 
and the bucketing was done in job two.  This meant that the bucketing in job 2 
partially undid the sorting of job 1, resulting in only some of the records 
showing up as deleted (since the records have to be written in the delta file 
in proper order).  The minimum number of reducers on which to apply the RS 
deduplication is pushed to 1 so that this optimization is used for even small 
queries.  

How is the changes to sort order of ROW__ID related?
That should never have been set to descending in the first place.  ROW__ID 
needs to be stored ascending to work properly.  I suspect it was a fluke of 
most of the qfile tests that they worked with this on.  (Actually Thejas asked 
at the time why this was necessary and rather than fixing it (which I should 
have done) I just said I didn't know.  Oops.)  

bq.  ReduceSinkDeDuplication.java change is not needed
What change?  I don't see any changes to that file in the patch.

> delete writes records in wrong order in some cases
> --
>
> Key: HIVE-8367
> URL: https://issues.apache.org/jira/browse/HIVE-8367
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8367.patch
>
>
> I have found one query with 10k records where you do:
> create table
> insert into table -- 10k records
> delete from table -- just some records
> The records in the delete delta are not ordered properly by rowid.
> I assume this applies to updates as well, but I haven't tested it yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

2014-10-08 Thread Pengcheng Xiong (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-8225:
--
Attachment: HIVE-8225.7.patch

resubmit patch

> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Pengcheng Xiong
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, 
> HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, 
> HIVE-8225.7.patch, HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, 
> HIVE-8225.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

2014-10-08 Thread Pengcheng Xiong (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-8225:
--
Status: Open  (was: Patch Available)

> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Pengcheng Xiong
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, 
> HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, 
> HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7932) It may cause NP exception when add accessed columns to ReadEntity

2014-10-08 Thread Prasad Mujumdar (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164320#comment-14164320
 ] 

Prasad Mujumdar commented on HIVE-7932:
---

+1
Looks fine to me. Thanks for adding the testcase.

> It may cause NP exception when add accessed columns to ReadEntity
> -
>
> Key: HIVE-7932
> URL: https://issues.apache.org/jira/browse/HIVE-7932
> Project: Hive
>  Issue Type: Bug
>Reporter: Xiaomeng Huang
>Assignee: Xiaomeng Huang
> Attachments: HIVE-7932.001.patch, HIVE-7932.002.patch
>
>
> {code}
> case TABLE:
>entity.getAccessedColumns().addAll(
>   tableToColumnAccessMap.get(entity.getTable().getCompleteName()));
> {code}
> if  tableToColumnAccessMap.get(entity.getTable().getCompleteName()) is null, 
> addAll(null) will throw null pointer exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

2014-10-08 Thread Sergey Shelukhin (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164316#comment-14164316
 ] 

Sergey Shelukhin commented on HIVE-8225:


is it possible to resubmit patch for HiveQA?


> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, 
> HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, 
> HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

2014-10-08 Thread Sergey Shelukhin (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8225:
---
Assignee: Pengcheng Xiong  (was: Sergey Shelukhin)

> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Pengcheng Xiong
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8225.1.patch, HIVE-8225.2.patch, HIVE-8225.3.patch, 
> HIVE-8225.4.patch, HIVE-8225.5.patch, HIVE-8225.6.patch, HIVE-8225.7.patch, 
> HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch, HIVE-8225.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8391) Comparion between TIMESTAMP and Integer types goes to STRING as "common comparison denominator" instead of a numeric type

2014-10-08 Thread Matt McCline (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-8391:
---
Attachment: (was: vectorization_7.q)

> Comparion between TIMESTAMP and Integer types goes to STRING as "common 
> comparison denominator" instead of a numeric type
> -
>
> Key: HIVE-8391
> URL: https://issues.apache.org/jira/browse/HIVE-8391
> Project: Hive
>  Issue Type: Bug
>  Components: CBO, Logical Optimizer
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Jason Dere
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8391.1.patch, HIVE-8391.2.patch
>
>
> Discovered while investigating why vectorization_7.q has different results 
> for non-vectorized [+CBO] and vectorized [+CBO].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (HIVE-7831) Research commented out unset in Utiltities [Spark Branch]

2014-10-08 Thread Xuefu Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang resolved HIVE-7831.
---
   Resolution: Not a Problem
Fix Version/s: spark-branch

> Research commented out unset in Utiltities [Spark Branch]
> -
>
> Key: HIVE-7831
> URL: https://issues.apache.org/jira/browse/HIVE-7831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
> Fix For: spark-branch
>
>
> We did the following in HIVE-7370
> {noformat}
> // TODO HIVE-7831
> // conf.unset(FsPermission.UMASK_LABEL);
> {noformat}
> We should understand that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7953) Investigate query failures (2)

2014-10-08 Thread Thomas Friedrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164301#comment-14164301
 ] 

Thomas Friedrich commented on HIVE-7953:


The two test cases 
auto_sortmerge_join_2 
auto_sortmerge_join_3
are from the minitez.query.files.shared list. 

Both fail with the same exception when running the statement 
select count(*) FROM bucket_big a JOIN bucket_small b ON a.key = b.key
I debugged the code and they fail in 

Daemon Thread [Executor task launch worker-1] (Suspended (exception 
HiveException)) 
MapOperator.process(Writable) line: 546 
SparkMapRecordHandler.processRow(Object, Object) line: 139  

HiveMapFunctionResultList.processNextRecord(Tuple2)
 line: 47   
HiveMapFunctionResultList.processNextRecord(Object) line: 28
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108   
Wrappers$JIteratorWrapper.hasNext() line: 41 
Iterator$class.foreach(Iterator, Function1) line: 727   

Wrappers$JIteratorWrapper(AbstractIterator).foreach(Function1) line: 
1157
RDD$$anonfun$foreach$1.apply(Iterator) line: 760 
RDD$$anonfun$foreach$1.apply(Object) line: 760  
SparkContext$$anonfun$runJob$3.apply(TaskContext, Iterator) line: 
1118   
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118 
ResultTask.runTask(TaskContext) line: 61   
ResultTask(Task).run(long) line: 56 
Executor$TaskRunner.run() line: 182 
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145  
ThreadPoolExecutor$Worker.run() line: 615   
Thread.run() line: 745  

The SparkException is:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 26.0 failed 1 times, most recent failure: Lost task 1.0 in stage 26.0 
(TID 27, localhost): java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row {"key":"100","value":"val_100","ds":"2008-04-08"}



> Investigate query failures (2)
> --
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, 
> but we should understand why they failed with an exception. This JIRA is to 
> look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external
>   0.33 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers
>   4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
> 11 sec  2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
> 0.65 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
>   4.7 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
>   2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2
>   5.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position 
> 1.5 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external
> 2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
> 11 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11   
> 5.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
>   10 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join 
> 5.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part
>  0.81 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 
>0.31 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1
> 0.26 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2  
> 0.73 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
> 8.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5  
> 0

[jira] [Commented] (HIVE-7831) Research commented out unset in Utiltities [Spark Branch]

2014-10-08 Thread Xuefu Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164302#comment-14164302
 ] 

Xuefu Zhang commented on HIVE-7831:
---

I don't see any commented out code any more. Since tests are passing w/o this 
change, I assume this change is not necessary. Closing this JIRA, therefore.

> Research commented out unset in Utiltities [Spark Branch]
> -
>
> Key: HIVE-7831
> URL: https://issues.apache.org/jira/browse/HIVE-7831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
> Fix For: spark-branch
>
>
> We did the following in HIVE-7370
> {noformat}
> // TODO HIVE-7831
> // conf.unset(FsPermission.UMASK_LABEL);
> {noformat}
> We should understand that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-8367) delete writes records in wrong order in some cases

2014-10-08 Thread Eugene Koifman (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164303#comment-14164303
 ] 

Eugene Koifman commented on HIVE-8367:
--

also, ReduceSinkDeDuplication.java change is not needed

> delete writes records in wrong order in some cases
> --
>
> Key: HIVE-8367
> URL: https://issues.apache.org/jira/browse/HIVE-8367
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8367.patch
>
>
> I have found one query with 10k records where you do:
> create table
> insert into table -- 10k records
> delete from table -- just some records
> The records in the delete delta are not ordered properly by rowid.
> I assume this applies to updates as well, but I haven't tested it yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7953) Investigate query failures (2)

2014-10-08 Thread Thomas Friedrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164299#comment-14164299
 ] 

Thomas Friedrich commented on HIVE-7953:


The following 4 tests are from the minimr.query.files list:
temp_table_external
load_hdfs_file_with_space_in_the_name
external_table_with_space_in_location_path
bucket_num_reducers

The 3 tests
temp_table_external
load_hdfs_file_with_space_in_the_name
external_table_with_space_in_location_path
create dirs in HDFS, but TestSparkCliDriver is not using HDFS. That's why the 
tests fail when creating a dir, for example mkdir -p 
hdfs:///tmp/temp_table_external
In the non-spark case, these tests are executed with the TestMinimrCliDriver 
which initializes Hadoop and created a mini-cluster
(see configuration in itests/qtest/pom.xml).

We may need to configure a TestSparkMinimrCliDriver in 
itests/qtest-spark/pom.xml if we want to run these test cases.

The test bucket_num_reducers doesn't use HDFS, and fails with a different error:
Exception: Number of MapReduce jobs is incorrect expected:<1> but was:<0>
junit.framework.AssertionFailedError: Number of MapReduce jobs is incorrect 
expected:<1> but was:<0>
at 
org.apache.hadoop.hive.ql.hooks.VerifyNumReducersHook.run(VerifyNumReducersHook.java:46)


> Investigate query failures (2)
> --
>
> Key: HIVE-7953
> URL: https://issues.apache.org/jira/browse/HIVE-7953
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests, 
> but we should understand why they failed with an exception. This JIRA is to 
> look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_temp_table_external
>   0.33 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers
>   4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_2
> 11 sec  2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
> 0.65 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_4
>   4.7 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_7
>   2.8 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_2
>   5.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_position 
> 1.5 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_18_part_external
> 2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
> 11 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_11   
> 5.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_8
>   10 sec  2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join 
> 5.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_empty_dyn_part
>  0.81 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact1 
>0.31 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_ddl1
> 0.26 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query2  
> 0.73 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
> 8.5 sec 2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query5  
> 0.34 sec2
>  org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_rcfile_bigdata   
> 0.93 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer
>  6.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact3 
>2.4 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_compact2 
>0.56 sec2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_partscan_1_23
>   3.1 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_list_bucket_dml_2 
>4.3 sec 2
>  
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_exim_15_external_part
> 3.2 sec 2
>  
> org.apache.hadoop.hiv

[jira] [Commented] (HIVE-8367) delete writes records in wrong order in some cases

2014-10-08 Thread Eugene Koifman (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164298#comment-14164298
 ] 

Eugene Koifman commented on HIVE-8367:
--

I think this needs more info.  What was the original query where the issue 
showed up?
What precisely was the problem and how does the RS deduplication change help?  
The explanation for the latter would be useful to add to the code where this 
setting is set.
How is the changes to sort order of ROW__ID related?

> delete writes records in wrong order in some cases
> --
>
> Key: HIVE-8367
> URL: https://issues.apache.org/jira/browse/HIVE-8367
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: HIVE-8367.patch
>
>
> I have found one query with 10k records where you do:
> create table
> insert into table -- 10k records
> delete from table -- just some records
> The records in the delete delta are not ordered properly by rowid.
> I assume this applies to updates as well, but I haven't tested it yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-7831) Research commented out unset in Utiltities [Spark Branch]

2014-10-08 Thread Xuefu Zhang (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-7831:
--
Summary: Research commented out unset in Utiltities [Spark Branch]  (was: 
Research commented out unset in Utiltities)

> Research commented out unset in Utiltities [Spark Branch]
> -
>
> Key: HIVE-7831
> URL: https://issues.apache.org/jira/browse/HIVE-7831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Brock Noland
>
> We did the following in HIVE-7370
> {noformat}
> // TODO HIVE-7831
> // conf.unset(FsPermission.UMASK_LABEL);
> {noformat}
> We should understand that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8391) Comparion between TIMESTAMP and Integer types goes to STRING as "common comparison denominator" instead of a numeric type

2014-10-08 Thread Jason Dere (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8391:
-
Status: Patch Available  (was: Open)

> Comparion between TIMESTAMP and Integer types goes to STRING as "common 
> comparison denominator" instead of a numeric type
> -
>
> Key: HIVE-8391
> URL: https://issues.apache.org/jira/browse/HIVE-8391
> Project: Hive
>  Issue Type: Bug
>  Components: CBO, Logical Optimizer
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Jason Dere
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8391.1.patch, HIVE-8391.2.patch, vectorization_7.q
>
>
> Discovered while investigating why vectorization_7.q has different results 
> for non-vectorized [+CBO] and vectorized [+CBO].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (HIVE-8391) Comparion between TIMESTAMP and Integer types goes to STRING as "common comparison denominator" instead of a numeric type

2014-10-08 Thread Jason Dere (JIRA)

 [ 
https://issues.apache.org/jira/browse/HIVE-8391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-8391:
-
Attachment: HIVE-8391.2.patch

Patch v2 allows timestamp and numeric types to be compared as double, without 
having to allow implicit conversions between timestamp and numeric. Also add q 
file test.

> Comparion between TIMESTAMP and Integer types goes to STRING as "common 
> comparison denominator" instead of a numeric type
> -
>
> Key: HIVE-8391
> URL: https://issues.apache.org/jira/browse/HIVE-8391
> Project: Hive
>  Issue Type: Bug
>  Components: CBO, Logical Optimizer
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Jason Dere
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8391.1.patch, HIVE-8391.2.patch, vectorization_7.q
>
>
> Discovered while investigating why vectorization_7.q has different results 
> for non-vectorized [+CBO] and vectorized [+CBO].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7776) enable sample10.q.[Spark Branch]

2014-10-08 Thread Xuefu Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164294#comment-14164294
 ] 

Xuefu Zhang commented on HIVE-7776:
---

Hi [~chengxiang li], Could you take a look at the above test result and see if 
the patch is ready to be committed? Thanks.

> enable sample10.q.[Spark Branch]
> 
>
> Key: HIVE-7776
> URL: https://issues.apache.org/jira/browse/HIVE-7776
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Chengxiang Li
>Assignee: Chengxiang Li
> Attachments: HIVE-7776.1-spark.patch, HIVE-7776.2-spark.patch, 
> HIVE-7776.3-spark.patch, HIVE-7776.3-spark.patch
>
>
> sample10.q contain dynamic partition operation, should enable this qtest 
> after hive on spark support dynamic partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


  1   2   3   >