date:20140331


[ 
https://issues.apache.org/jira/browse/HIVE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955195#comment-13955195
 ] 

Xuefu Zhang commented on HIVE-6758:
---

Thank you so much for providing this useful information, [~qwertymaniac]. 
Please feel free to assign this JIRA to yourself if you'd like to working on 
this.

 Beeline doesn't work with -e option when started in background
 --

 Key: HIVE-6758
 URL: https://issues.apache.org/jira/browse/HIVE-6758
 Project: Hive
  Issue Type: Improvement
  Components: CLI
Affects Versions: 0.11.0
Reporter: Johndee Burks
Assignee: Xuefu Zhang

 In hive CLI you could easily integrate its use into a script and back ground 
 the process like this: 
 hive -e some query 
 Beeline does not run when you do the same even with the -f switch. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6570) Hive variable substitution does not work with the source command

2014-03-31 Thread Anthony Hsu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955374#comment-13955374
 ] 

Anthony Hsu commented on HIVE-6570:
---

[~leftylev] - Thanks for the instructions!
[~xuefuz] - Thanks for committing this!

 Hive variable substitution does not work with the source command
 --

 Key: HIVE-6570
 URL: https://issues.apache.org/jira/browse/HIVE-6570
 Project: Hive
  Issue Type: Bug
Reporter: Anthony Hsu
Assignee: Anthony Hsu
 Fix For: 0.14.0

 Attachments: HIVE-6570.1.patch


 The following does not work:
 {code}
 source ${hivevar:test-dir}/test.q;
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6786) Off by one error in ORC PPD

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955464#comment-13955464
 ] 

Sergey Shelukhin commented on HIVE-6786:


+1

 Off by one error in ORC PPD 
 

 Key: HIVE-6786
 URL: https://issues.apache.org/jira/browse/HIVE-6786
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Gopal V
Assignee: Prasanth J
Priority: Critical
 Fix For: 0.13.0

 Attachments: HIVE-6786.1.patch


 Turning on ORC PPD makes split computation fail for a 10Tb benchmark.
 Narrowed down to the following code fragment
 https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L757
 {code}
 includeStripe[i] = (i  stripeStats.size()) ||
 isStripeSatisfyPredicate(stripeStats.get(i), sarg,
  filterColumns);
 {code}
 I would guess that should be a =, but [~prasanth_j], can you comment if that 
 is the right fix?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6642) Query fails to vectorize when a non string partition column is part of the query expression


 [ 
https://issues.apache.org/jira/browse/HIVE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6642:


Status: Patch Available  (was: Open)

 Query fails to vectorize when a non string partition column is part of the 
 query expression
 ---

 Key: HIVE-6642
 URL: https://issues.apache.org/jira/browse/HIVE-6642
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-6642-2.patch, HIVE-6642-3.patch, HIVE-6642-4.patch, 
 HIVE-6642.1.patch, HIVE-6642.5.patch, HIVE-6642.6.patch, HIVE-6642.7.patch


 drop table if exists alltypesorc_part;
 CREATE TABLE alltypesorc_part (
 ctinyint tinyint,
 csmallint smallint,
 cint int,
 cbigint bigint,
 cfloat float,
 cdouble double,
 cstring1 string,
 cstring2 string,
 ctimestamp1 timestamp,
 ctimestamp2 timestamp,
 cboolean1 boolean,
 cboolean2 boolean) partitioned by (ds int) STORED AS ORC;
 insert overwrite table alltypesorc_part partition (ds=2011) select * from 
 alltypesorc limit 100;
 insert overwrite table alltypesorc_part partition (ds=2012) select * from 
 alltypesorc limit 200;
 explain select *
 from (select ds from alltypesorc_part) t1,
  alltypesorc t2
 where t1.ds = t2.cint
 order by t2.ctimestamp1
 limit 100;
 The above query fails to vectorize because (select ds from alltypesorc_part) 
 t1 returns a string column and the join equality on t2 is performed on an int 
 column. The correct output when vectorization is turned on should be:
 STAGE DEPENDENCIES:
   Stage-5 is a root stage
   Stage-2 depends on stages: Stage-5
   Stage-0 is a root stage
 STAGE PLANS:
   Stage: Stage-5
 Map Reduce Local Work
   Alias - Map Local Tables:
 t1:alltypesorc_part
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 t1:alltypesorc_part
   TableScan
 alias: alltypesorc_part
 Statistics: Num rows: 300 Data size: 62328 Basic stats: COMPLETE 
 Column stats: COMPLETE
 Select Operator
   expressions: ds (type: int)
   outputColumnNames: _col0
   Statistics: Num rows: 300 Data size: 1200 Basic stats: COMPLETE 
 Column stats: COMPLETE
   HashTable Sink Operator
 condition expressions:
   0 {_col0}
   1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} 
 {cdouble} {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} 
 {cboolean2}
 keys:
   0 _col0 (type: int)
   1 cint (type: int)
   Stage: Stage-2
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: t2
 Statistics: Num rows: 3536 Data size: 1131711 Basic stats: 
 COMPLETE Column stats: NONE
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {_col0}
 1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
 {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
   keys:
 0 _col0 (type: int)
 1 cint (type: int)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
 _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 3889 Data size: 1244882 Basic stats: 
 COMPLETE Column stats: NONE
   Filter Operator
 predicate: (_col0 = _col3) (type: boolean)
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 Select Operator
   expressions: _col0 (type: int), _col1 (type: tinyint), 
 _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), _col5 (type: 
 float), _col6 (type: double), _col7 (type: string), _col8 (type: string), 
 _col\
 9 (type: timestamp), _col10 (type: timestamp), _col11 (type: boolean), _col12 
 (type: boolean)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, 
 _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
   Reduce Output Operator
 key expressions: _col9 (type: timestamp)
 sort order: +
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 value expressions: _col0 (type: int), _col1 (type: 
 tinyint), _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), 
 _col5 (type: float),

[jira] [Updated] (HIVE-6642) Query fails to vectorize when a non string partition column is part of the query expression

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6642:


Status: Open  (was: Patch Available)

 Query fails to vectorize when a non string partition column is part of the 
 query expression
 ---

 Key: HIVE-6642
 URL: https://issues.apache.org/jira/browse/HIVE-6642
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-6642-2.patch, HIVE-6642-3.patch, HIVE-6642-4.patch, 
 HIVE-6642.1.patch, HIVE-6642.5.patch, HIVE-6642.6.patch, HIVE-6642.7.patch


 drop table if exists alltypesorc_part;
 CREATE TABLE alltypesorc_part (
 ctinyint tinyint,
 csmallint smallint,
 cint int,
 cbigint bigint,
 cfloat float,
 cdouble double,
 cstring1 string,
 cstring2 string,
 ctimestamp1 timestamp,
 ctimestamp2 timestamp,
 cboolean1 boolean,
 cboolean2 boolean) partitioned by (ds int) STORED AS ORC;
 insert overwrite table alltypesorc_part partition (ds=2011) select * from 
 alltypesorc limit 100;
 insert overwrite table alltypesorc_part partition (ds=2012) select * from 
 alltypesorc limit 200;
 explain select *
 from (select ds from alltypesorc_part) t1,
  alltypesorc t2
 where t1.ds = t2.cint
 order by t2.ctimestamp1
 limit 100;
 The above query fails to vectorize because (select ds from alltypesorc_part) 
 t1 returns a string column and the join equality on t2 is performed on an int 
 column. The correct output when vectorization is turned on should be:
 STAGE DEPENDENCIES:
   Stage-5 is a root stage
   Stage-2 depends on stages: Stage-5
   Stage-0 is a root stage
 STAGE PLANS:
   Stage: Stage-5
 Map Reduce Local Work
   Alias - Map Local Tables:
 t1:alltypesorc_part
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 t1:alltypesorc_part
   TableScan
 alias: alltypesorc_part
 Statistics: Num rows: 300 Data size: 62328 Basic stats: COMPLETE 
 Column stats: COMPLETE
 Select Operator
   expressions: ds (type: int)
   outputColumnNames: _col0
   Statistics: Num rows: 300 Data size: 1200 Basic stats: COMPLETE 
 Column stats: COMPLETE
   HashTable Sink Operator
 condition expressions:
   0 {_col0}
   1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} 
 {cdouble} {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} 
 {cboolean2}
 keys:
   0 _col0 (type: int)
   1 cint (type: int)
   Stage: Stage-2
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: t2
 Statistics: Num rows: 3536 Data size: 1131711 Basic stats: 
 COMPLETE Column stats: NONE
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {_col0}
 1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
 {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
   keys:
 0 _col0 (type: int)
 1 cint (type: int)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
 _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 3889 Data size: 1244882 Basic stats: 
 COMPLETE Column stats: NONE
   Filter Operator
 predicate: (_col0 = _col3) (type: boolean)
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 Select Operator
   expressions: _col0 (type: int), _col1 (type: tinyint), 
 _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), _col5 (type: 
 float), _col6 (type: double), _col7 (type: string), _col8 (type: string), 
 _col\
 9 (type: timestamp), _col10 (type: timestamp), _col11 (type: boolean), _col12 
 (type: boolean)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, 
 _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
   Reduce Output Operator
 key expressions: _col9 (type: timestamp)
 sort order: +
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 value expressions: _col0 (type: int), _col1 (type: 
 tinyint), _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), 
 _col5 (type: float),

[jira] [Updated] (HIVE-6642) Query fails to vectorize when a non string partition column is part of the query expression

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6642:


Attachment: HIVE-6642.7.patch

 Query fails to vectorize when a non string partition column is part of the 
 query expression
 ---

 Key: HIVE-6642
 URL: https://issues.apache.org/jira/browse/HIVE-6642
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-6642-2.patch, HIVE-6642-3.patch, HIVE-6642-4.patch, 
 HIVE-6642.1.patch, HIVE-6642.5.patch, HIVE-6642.6.patch, HIVE-6642.7.patch


 drop table if exists alltypesorc_part;
 CREATE TABLE alltypesorc_part (
 ctinyint tinyint,
 csmallint smallint,
 cint int,
 cbigint bigint,
 cfloat float,
 cdouble double,
 cstring1 string,
 cstring2 string,
 ctimestamp1 timestamp,
 ctimestamp2 timestamp,
 cboolean1 boolean,
 cboolean2 boolean) partitioned by (ds int) STORED AS ORC;
 insert overwrite table alltypesorc_part partition (ds=2011) select * from 
 alltypesorc limit 100;
 insert overwrite table alltypesorc_part partition (ds=2012) select * from 
 alltypesorc limit 200;
 explain select *
 from (select ds from alltypesorc_part) t1,
  alltypesorc t2
 where t1.ds = t2.cint
 order by t2.ctimestamp1
 limit 100;
 The above query fails to vectorize because (select ds from alltypesorc_part) 
 t1 returns a string column and the join equality on t2 is performed on an int 
 column. The correct output when vectorization is turned on should be:
 STAGE DEPENDENCIES:
   Stage-5 is a root stage
   Stage-2 depends on stages: Stage-5
   Stage-0 is a root stage
 STAGE PLANS:
   Stage: Stage-5
 Map Reduce Local Work
   Alias - Map Local Tables:
 t1:alltypesorc_part
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 t1:alltypesorc_part
   TableScan
 alias: alltypesorc_part
 Statistics: Num rows: 300 Data size: 62328 Basic stats: COMPLETE 
 Column stats: COMPLETE
 Select Operator
   expressions: ds (type: int)
   outputColumnNames: _col0
   Statistics: Num rows: 300 Data size: 1200 Basic stats: COMPLETE 
 Column stats: COMPLETE
   HashTable Sink Operator
 condition expressions:
   0 {_col0}
   1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} 
 {cdouble} {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} 
 {cboolean2}
 keys:
   0 _col0 (type: int)
   1 cint (type: int)
   Stage: Stage-2
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: t2
 Statistics: Num rows: 3536 Data size: 1131711 Basic stats: 
 COMPLETE Column stats: NONE
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {_col0}
 1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
 {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
   keys:
 0 _col0 (type: int)
 1 cint (type: int)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
 _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 3889 Data size: 1244882 Basic stats: 
 COMPLETE Column stats: NONE
   Filter Operator
 predicate: (_col0 = _col3) (type: boolean)
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 Select Operator
   expressions: _col0 (type: int), _col1 (type: tinyint), 
 _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), _col5 (type: 
 float), _col6 (type: double), _col7 (type: string), _col8 (type: string), 
 _col\
 9 (type: timestamp), _col10 (type: timestamp), _col11 (type: boolean), _col12 
 (type: boolean)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, 
 _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
   Reduce Output Operator
 key expressions: _col9 (type: timestamp)
 sort order: +
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 value expressions: _col0 (type: int), _col1 (type: 
 tinyint), _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), 
 _col5 (type: float), _col6

[jira] [Created] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory

Xuefu Zhang created HIVE-6793:
-

 Summary: DDLSemanticAnalyzer.analyzeShowRoles() should use 
HiveAuthorizationTaskFactory
 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
related task creations such as for analyzeShowRolePrincipals(). This JIRA is to 
make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory


 [ 
https://issues.apache.org/jira/browse/HIVE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6793:
--

Component/s: Query Processor

 DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
 --

 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Query Processor
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang

 Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
 HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
 related task creations such as for analyzeShowRolePrincipals(). This JIRA is 
 to make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5835) Null pointer exception in DeleteDelegator in templeton code


[ 
https://issues.apache.org/jira/browse/HIVE-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955490#comment-13955490
 ] 

Harish Butani commented on HIVE-5835:
-

+1 for 0.13

 Null pointer exception in DeleteDelegator in templeton code 
 

 Key: HIVE-5835
 URL: https://issues.apache.org/jira/browse/HIVE-5835
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-5835.1.patch, HIVE-5835.1.patch


 The following NPE is possible with the current implementation:
 ERROR | 13 Nov 2013 08:01:04,292 | 
 org.apache.hcatalog.templeton.CatchallExceptionMapper | 
 java.lang.NullPointerException
 at org.apache.hcatalog.templeton.tool.JobState.getChildren(JobState.java:180)
 at org.apache.hcatalog.templeton.DeleteDelegator.run(DeleteDelegator.java:51)
 at org.apache.hcatalog.templeton.Server.deleteJobId(Server.java:849)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
 com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 at 
 com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
 at 
 com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at 
 com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 at 
 com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
 com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at 
 com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
 com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
 at 
 com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
 at 
 com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
 at 
 com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
 at 
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
 at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:382)
 at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
 at 
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
 at 
 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
 at 
 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
 at 
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
 at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
 at 
 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
 at org.eclipse.jetty.server.Server.handle(Server.java:349)
 at 
 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
 at 
 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
 at 
 org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
 at 
 org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
 at 
 org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
 at 
 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
 at

[jira] [Updated] (HIVE-5835) Null pointer exception in DeleteDelegator in templeton code


 [ 
https://issues.apache.org/jira/browse/HIVE-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5835:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk and 0.13
thanks Hari and Thejas.

 Null pointer exception in DeleteDelegator in templeton code 
 

 Key: HIVE-5835
 URL: https://issues.apache.org/jira/browse/HIVE-5835
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-5835.1.patch, HIVE-5835.1.patch


 The following NPE is possible with the current implementation:
 ERROR | 13 Nov 2013 08:01:04,292 | 
 org.apache.hcatalog.templeton.CatchallExceptionMapper | 
 java.lang.NullPointerException
 at org.apache.hcatalog.templeton.tool.JobState.getChildren(JobState.java:180)
 at org.apache.hcatalog.templeton.DeleteDelegator.run(DeleteDelegator.java:51)
 at org.apache.hcatalog.templeton.Server.deleteJobId(Server.java:849)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at 
 com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
 at 
 com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
 at 
 com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
 at 
 com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
 at 
 com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
 com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
 at 
 com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
 at 
 com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
 at 
 com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
 at 
 com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
 at 
 com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
 at 
 com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
 at 
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
 at 
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:382)
 at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
 at 
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
 at 
 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
 at 
 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
 at 
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
 at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
 at 
 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
 at org.eclipse.jetty.server.Server.handle(Server.java:349)
 at 
 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
 at 
 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
 at 
 org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
 at 
 org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
 at 
 org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
 at

[jira] [Commented] (HIVE-6779) Hive cli may get into inconsistent state when Ctrl-C is hit on hadoop2


[ 
https://issues.apache.org/jira/browse/HIVE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955537#comment-13955537
 ] 

Ashutosh Chauhan commented on HIVE-6779:


To add more context: Important part of patch is removal of interrupting of main 
thread for SignalHandler. In hadop-1 java io with sockets were used to 
communicate between client and server. So, interrupting main thread wasn't an 
issue. However,  Hadoop-2 uses java nio based interruptible channels, which 
throws up if current thread is interrupted. 

 Hive cli may get into inconsistent state when Ctrl-C is hit on hadoop2
 --

 Key: HIVE-6779
 URL: https://issues.apache.org/jira/browse/HIVE-6779
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
 Attachments: HIVE-6779.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory


 [ 
https://issues.apache.org/jira/browse/HIVE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6793:
--

Attachment: HIVE-6793.patch

 DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
 --

 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Query Processor
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Attachments: HIVE-6793.patch


 Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
 HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
 related task creations such as for analyzeShowRolePrincipals(). This JIRA is 
 to make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory


 [ 
https://issues.apache.org/jira/browse/HIVE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6793:
--

Status: Patch Available  (was: Open)

 DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
 --

 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Query Processor
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Attachments: HIVE-6793.patch


 Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
 HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
 related task creations such as for analyzeShowRolePrincipals(). This JIRA is 
 to make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 19856: HIVE-6793: DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory

2014-03-31 Thread Xuefu Zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19856/
---

Review request for hive.


Bugs: HIVE-6793
https://issues.apache.org/jira/browse/HIVE-6793


Repository: hive-git


Description
---

Please refer to the JIRA description.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 117295a 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactory.java
 cc84731 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
 0354604 

Diff: https://reviews.apache.org/r/19856/diff/


Testing
---

Pure refactoring. No new test cases are added. Patch should be okay if 
pre-commit test passes.


Thanks,

Xuefu Zhang

Re: Review Request 19856: HIVE-6793: DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory

2014-03-31 Thread Brock Noland


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19856/#review39089
---

Ship it!


Ship It!

- Brock Noland


On March 31, 2014, 7:07 p.m., Xuefu Zhang wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/19856/
 ---
 
 (Updated March 31, 2014, 7:07 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-6793
 https://issues.apache.org/jira/browse/HIVE-6793
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Please refer to the JIRA description.
 
 
 Diffs
 -
 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 117295a 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactory.java
  cc84731 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
  0354604 
 
 Diff: https://reviews.apache.org/r/19856/diff/
 
 
 Testing
 ---
 
 Pure refactoring. No new test cases are added. Patch should be okay if 
 pre-commit test passes.
 
 
 Thanks,
 
 Xuefu Zhang

[jira] [Created] (HIVE-6794) select * from parquet hive table containing map columns runs into exception

2014-03-31 Thread Tongjie Chen (JIRA)

Tongjie Chen created HIVE-6794:
--

 Summary: select * from parquet hive table containing map columns 
runs into exception
 Key: HIVE-6794
 URL: https://issues.apache.org/jira/browse/HIVE-6794
 Project: Hive
  Issue Type: Bug
  Components: File Formats, Serializers/Deserializers
Affects Versions: 0.13.0
Reporter: Tongjie Chen


select * from parquet hive table containing map columns run into exception 
FAILED: RuntimeException java.lang.ClassCastException: 
parquet.hive.serde.DeepParquetHiveMapInspector cannot be cast to 
parquet.hive.serde.StandardParquetHiveMapInspector

To reproduce, use the following steps (both regular_map_table and 
parquet_map_table contains schema c1 , c2 int):

hive insert overwrite table parquet_map_table select * from regular_map_table;
hive select * from parquet_map_table; (the first query to select * works fine )

OK
{key1:value1} 1
{key2:value2} 2
{key3:value3} 3
Time taken: 2.669 seconds, Fetched: 3 row(s)
hive select * from parquet_map_table; (the second and all subsequent exact 
same query breaks)

FAILED: RuntimeException java.lang.ClassCastException: 
parquet.hive.serde.DeepParquetHiveMapInspector cannot be cast to 
parquet.hive.serde.StandardParquetHiveMapInspector

Interestingly select c1 from parquet_map_column query works fine though.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6783) Incompatible schema for maps between parquet-hive and parquet-pig

2014-03-31 Thread Tongjie Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955563#comment-13955563
 ] 

Tongjie Chen commented on HIVE-6783:


I think it should be backward compatible, I am able to read existing stored 
maps with this fix.

 Incompatible schema for maps between parquet-hive and parquet-pig
 -

 Key: HIVE-6783
 URL: https://issues.apache.org/jira/browse/HIVE-6783
 Project: Hive
  Issue Type: Bug
  Components: File Formats
Affects Versions: 0.13.0
Reporter: Tongjie Chen
 Fix For: 0.13.0

 Attachments: HIVE-6783.1.patch.txt, HIVE-6783.2.patch.txt, 
 HIVE-6783.3.patch.txt, HIVE-6783.4.patch.txt


 see also in following parquet issue:
 https://github.com/Parquet/parquet-mr/issues/290
 The schema written for maps isn't compatible between hive and pig. This means 
 any files written in one cannot be properly read in the other.
 More specifically,  for the same map column c1, parquet-pig generates schema:
 message pig_schema {
   optional group c1 (MAP) {
 repeated group map (MAP_KEY_VALUE) {
   required binary key (UTF8);
   optional binary value;
 }   
   }
 }
 while parquet-hive generates schema:
 message hive_schema {
optional group c1 (MAP_KEY_VALUE) {
  repeated group map {
required binary key;
optional binary value;
}
  }
 }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955570#comment-13955570
]

Pala M Muthaia commented on HIVE-6131:
--

[~ashutoshc], any thoughts on this?

New columns after table alter result in null values despite data

Key: HIVE-6131
URL: https://issues.apache.org/jira/browse/HIVE-6131
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: James Vaughan
Priority: Minor
Attachments: HIVE-6131.1.patch.txt

Hi folks,
I found and verified a bug on our CDH 4.0.3 install of Hive when adding
columns to tables with Partitions using 'REPLACE COLUMNS'. I dug through the
Jira a little bit and didn't see anything for it so hopefully this isn't just
noise on the radar.
Basically, when you alter a table with partitions and then reupload data to
that partition, it doesn't seem to recognize the extra data that actually
exists in HDFS- as in, returns NULL values on the new column despite having
the data and recognizing the new column in the metadata.
Here's some steps to reproduce using a basic table:
1. Run this hive command: CREATE TABLE jvaughan_test (col1 string)
partitioned by (day string);
2. Create a simple file on the system with a couple of entries, something
like hi and hi2 separated by newlines.
3. Run this hive command, pointing it at the file: LOAD DATA LOCAL INPATH
'FILEDIR' OVERWRITE INTO TABLE jvaughan_test PARTITION (day = '2014-01-02');
4. Confirm the data with: SELECT * FROM jvaughan_test WHERE day =
'2014-01-02';
5. Alter the column definitions: ALTER TABLE jvaughan_test REPLACE COLUMNS
(col1 string, col2 string);
6. Edit your file and add a second column using the default separator
(ctrl+v, then ctrl+a in Vim) and add two more entries, such as hi3 on the
first row and hi4 on the second
7. Run step 3 again
8. Check the data again like in step 4
For me, this is the results that get returned:
hive select * from jvaughan_test where day = '2014-01-01';
OK
hiNULL2014-01-02
hi2 NULL2014-01-02
This is despite the fact that there is data in the file stored by the
partition in HDFS.
Let me know if you need any other information. The only workaround for me
currently is to drop partitions for any I'm replacing data in and THEN
reupload the new data file.
Thanks,
-James

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-4329) HCatalog clients can't write to AvroSerde backed tables

2014-03-31 Thread David Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955594#comment-13955594
 ] 

David Chen commented on HIVE-4329:
--

I am running into this issue as well. If no one is currently working on this 
ticket, I would like to pick it up.

 HCatalog clients can't write to AvroSerde backed tables
 ---

 Key: HIVE-4329
 URL: https://issues.apache.org/jira/browse/HIVE-4329
 Project: Hive
  Issue Type: Bug
  Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.10.0
 Environment: discovered in Pig, but it looks like the root cause 
 impacts all non-Hive users
Reporter: Sean Busbey

 Attempting to write to a HCatalog defined table backed by the AvroSerde fails 
 with the following stacktrace:
 {code}
 java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be 
 cast to org.apache.hadoop.io.LongWritable
   at 
 org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84)
   at 
 org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253)
   at 
 org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
   at 
 org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242)
   at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52)
   at 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
   at 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
   at 
 org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
   at 
 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
 {code}
 The proximal cause of this failure is that the AvroContainerOutputFormat's 
 signature mandates a LongWritable key and HCat's FileRecordWriterContainer 
 forces a NullWritable. I'm not sure of a general fix, other than redefining 
 HiveOutputFormat to mandate a WritableComparable.
 It looks like accepting WritableComparable is what's done in the other Hive 
 OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also 
 be changed, since it's ignoring the key. That way fixing things so 
 FileRecordWriterContainer can always use NullWritable could get spun into a 
 different issue?
 The underlying cause for failure to write to AvroSerde tables is that 
 AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so 
 fixing the above will just push the failure into the placeholder RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6642) Query fails to vectorize when a non string partition column is part of the query expression

2014-03-31 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955693#comment-13955693
 ] 

Hive QA commented on HIVE-6642:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12637892/HIVE-6642.7.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5513 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2051/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2051/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12637892

 Query fails to vectorize when a non string partition column is part of the 
 query expression
 ---

 Key: HIVE-6642
 URL: https://issues.apache.org/jira/browse/HIVE-6642
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-6642-2.patch, HIVE-6642-3.patch, HIVE-6642-4.patch, 
 HIVE-6642.1.patch, HIVE-6642.5.patch, HIVE-6642.6.patch, HIVE-6642.7.patch


 drop table if exists alltypesorc_part;
 CREATE TABLE alltypesorc_part (
 ctinyint tinyint,
 csmallint smallint,
 cint int,
 cbigint bigint,
 cfloat float,
 cdouble double,
 cstring1 string,
 cstring2 string,
 ctimestamp1 timestamp,
 ctimestamp2 timestamp,
 cboolean1 boolean,
 cboolean2 boolean) partitioned by (ds int) STORED AS ORC;
 insert overwrite table alltypesorc_part partition (ds=2011) select * from 
 alltypesorc limit 100;
 insert overwrite table alltypesorc_part partition (ds=2012) select * from 
 alltypesorc limit 200;
 explain select *
 from (select ds from alltypesorc_part) t1,
  alltypesorc t2
 where t1.ds = t2.cint
 order by t2.ctimestamp1
 limit 100;
 The above query fails to vectorize because (select ds from alltypesorc_part) 
 t1 returns a string column and the join equality on t2 is performed on an int 
 column. The correct output when vectorization is turned on should be:
 STAGE DEPENDENCIES:
   Stage-5 is a root stage
   Stage-2 depends on stages: Stage-5
   Stage-0 is a root stage
 STAGE PLANS:
   Stage: Stage-5
 Map Reduce Local Work
   Alias - Map Local Tables:
 t1:alltypesorc_part
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 t1:alltypesorc_part
   TableScan
 alias: alltypesorc_part
 Statistics: Num rows: 300 Data size: 62328 Basic stats: COMPLETE 
 Column stats: COMPLETE
 Select Operator
   expressions: ds (type: int)
   outputColumnNames: _col0
   Statistics: Num rows: 300 Data size: 1200 Basic stats: COMPLETE 
 Column stats: COMPLETE
   HashTable Sink Operator
 condition expressions:
   0 {_col0}
   1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} 
 {cdouble} {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} 
 {cboolean2}
 keys:
   0 _col0 (type: int)
   1 cint (type: int)
   Stage: Stage-2
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: t2
 Statistics: Num rows: 3536 Data size: 1131711 Basic stats: 
 COMPLETE Column stats: NONE
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {_col0}
 1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
 {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
   keys:
 0 _col0 (type: int)
 1 cint (type: int)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
 _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 3889 Data size: 1244882 Basic stats: 
 COMPLETE Column stats: NONE
   Filter Operator
 predicate: (_col0 = _col3) (type: boolean)
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 Select Operator
   expressions: _col0 (type: int), _col1 (type: tinyint), 
 _col2 (type: smallint), _col3 (type: int), _col4 (type:

[jira] [Commented] (HIVE-6779) Hive cli may get into inconsistent state when Ctrl-C is hit on hadoop2

2014-03-31 Thread Jason Dere (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955695#comment-13955695
 ] 

Jason Dere commented on HIVE-6779:
--

+1

 Hive cli may get into inconsistent state when Ctrl-C is hit on hadoop2
 --

 Key: HIVE-6779
 URL: https://issues.apache.org/jira/browse/HIVE-6779
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
 Attachments: HIVE-6779.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6775) Altering partition column type should not be allowed when there is existing partitions for which column types cannot be casted to the new type.


 [ 
https://issues.apache.org/jira/browse/HIVE-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan reassigned HIVE-6775:
---

Assignee: Hari Sankar Sivarama Subramaniyan

 Altering partition column type should not be allowed when there is existing 
 partitions for which column types cannot be casted to the new type.
 ---

 Key: HIVE-6775
 URL: https://issues.apache.org/jira/browse/HIVE-6775
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6778) ql/src/test/queries/clientpositive/pcr.q covers the test which generate 1.0 =1 predicate in partition pruner.

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6778:


Assignee: Harish Butani

 ql/src/test/queries/clientpositive/pcr.q covers the test which generate 1.0 
 =1 predicate in partition pruner. 
 --

 Key: HIVE-6778
 URL: https://issues.apache.org/jira/browse/HIVE-6778
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Harish Butani

 select key, value, ds from pcr_foo where (ds % 2 == 1);
 ql/src/test/queries/clientpositive/pcr.q
 The test generates 1.0==1 predicate in the pruner which cannot be evaluated 
 since a double cannot be converted to int.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6795) metastore initialization should add default roles with default, SBA

Thejas M Nair created HIVE-6795:
---

 Summary: metastore initialization should add default roles with 
default, SBA
 Key: HIVE-6795
 URL: https://issues.apache.org/jira/browse/HIVE-6795
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair


Hiveserver2 running sql standard authorization can connect to a metastore 
running storage based authorization. Currently metastore is not adding the 
standard roles to the db in such cases.
It would be better to add them in these cases as well.





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6642) Query fails to vectorize when a non string partition column is part of the query expression


 [ 
https://issues.apache.org/jira/browse/HIVE-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6642:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to trunk and 0.13
thanks Hari.

 Query fails to vectorize when a non string partition column is part of the 
 query expression
 ---

 Key: HIVE-6642
 URL: https://issues.apache.org/jira/browse/HIVE-6642
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan
 Fix For: 0.13.0

 Attachments: HIVE-6642-2.patch, HIVE-6642-3.patch, HIVE-6642-4.patch, 
 HIVE-6642.1.patch, HIVE-6642.5.patch, HIVE-6642.6.patch, HIVE-6642.7.patch


 drop table if exists alltypesorc_part;
 CREATE TABLE alltypesorc_part (
 ctinyint tinyint,
 csmallint smallint,
 cint int,
 cbigint bigint,
 cfloat float,
 cdouble double,
 cstring1 string,
 cstring2 string,
 ctimestamp1 timestamp,
 ctimestamp2 timestamp,
 cboolean1 boolean,
 cboolean2 boolean) partitioned by (ds int) STORED AS ORC;
 insert overwrite table alltypesorc_part partition (ds=2011) select * from 
 alltypesorc limit 100;
 insert overwrite table alltypesorc_part partition (ds=2012) select * from 
 alltypesorc limit 200;
 explain select *
 from (select ds from alltypesorc_part) t1,
  alltypesorc t2
 where t1.ds = t2.cint
 order by t2.ctimestamp1
 limit 100;
 The above query fails to vectorize because (select ds from alltypesorc_part) 
 t1 returns a string column and the join equality on t2 is performed on an int 
 column. The correct output when vectorization is turned on should be:
 STAGE DEPENDENCIES:
   Stage-5 is a root stage
   Stage-2 depends on stages: Stage-5
   Stage-0 is a root stage
 STAGE PLANS:
   Stage: Stage-5
 Map Reduce Local Work
   Alias - Map Local Tables:
 t1:alltypesorc_part
   Fetch Operator
 limit: -1
   Alias - Map Local Operator Tree:
 t1:alltypesorc_part
   TableScan
 alias: alltypesorc_part
 Statistics: Num rows: 300 Data size: 62328 Basic stats: COMPLETE 
 Column stats: COMPLETE
 Select Operator
   expressions: ds (type: int)
   outputColumnNames: _col0
   Statistics: Num rows: 300 Data size: 1200 Basic stats: COMPLETE 
 Column stats: COMPLETE
   HashTable Sink Operator
 condition expressions:
   0 {_col0}
   1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} 
 {cdouble} {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} 
 {cboolean2}
 keys:
   0 _col0 (type: int)
   1 cint (type: int)
   Stage: Stage-2
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: t2
 Statistics: Num rows: 3536 Data size: 1131711 Basic stats: 
 COMPLETE Column stats: NONE
 Map Join Operator
   condition map:
Inner Join 0 to 1
   condition expressions:
 0 {_col0}
 1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
 {cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
   keys:
 0 _col0 (type: int)
 1 cint (type: int)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
 _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 3889 Data size: 1244882 Basic stats: 
 COMPLETE Column stats: NONE
   Filter Operator
 predicate: (_col0 = _col3) (type: boolean)
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 Select Operator
   expressions: _col0 (type: int), _col1 (type: tinyint), 
 _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), _col5 (type: 
 float), _col6 (type: double), _col7 (type: string), _col8 (type: string), 
 _col\
 9 (type: timestamp), _col10 (type: timestamp), _col11 (type: boolean), _col12 
 (type: boolean)
   outputColumnNames: _col0, _col1, _col2, _col3, _col4, 
 _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
   Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
   Reduce Output Operator
 key expressions: _col9 (type: timestamp)
 sort order: +
 Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
 COMPLETE Column stats: NONE
 value expressions: _col0 (type: int), _col1 (type: 
 tinyint), _col2 (type: smallint), _col3 (type: int), _col4 (type:

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955727#comment-13955727
]

Pala M Muthaia commented on HIVE-6131:
--

Thanks, will do [~ashutoshc]. However, i need a login with apache jenkins.
Could you or somebody else add a login for me?

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Szehon Ho reassigned HIVE-6131:
---

Assignee: Szehon Ho

New columns after table alter result in null values despite data

Key: HIVE-6131
URL: https://issues.apache.org/jira/browse/HIVE-6131
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: James Vaughan
Assignee: Szehon Ho
Priority: Minor
Attachments: HIVE-6131.1.patch.txt

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955734#comment-13955734
]

Szehon Ho commented on HIVE-6131:
-

Hi Pala, you can just re-upload the same patch again. Jenkins job will pick it
up automatically. I think the first patch you uploaded got missed by the
jenkins job during an outage.

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Szehon Ho reassigned HIVE-6131:
---

Assignee: (was: Szehon Ho)

Sorry I assigned to myself accidentally while commenting, please assign back to
yourself.

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pala M Muthaia updated HIVE-6131:
-

Attachment: (was: HIVE-6131.1.patch.txt)

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pala M Muthaia updated HIVE-6131:
-

Attachment: HIVE-6131.1.patch.txt

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-4329) HCatalog clients can't write to AvroSerde backed tables

2014-03-31 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955736#comment-13955736
 ] 

Sean Busbey commented on HIVE-4329:
---

I don't think anyone's working on it. I'd be happy to review once you post a 
patch.

 HCatalog clients can't write to AvroSerde backed tables
 ---

 Key: HIVE-4329
 URL: https://issues.apache.org/jira/browse/HIVE-4329
 Project: Hive
  Issue Type: Bug
  Components: HCatalog, Serializers/Deserializers
Affects Versions: 0.10.0
 Environment: discovered in Pig, but it looks like the root cause 
 impacts all non-Hive users
Reporter: Sean Busbey

 Attempting to write to a HCatalog defined table backed by the AvroSerde fails 
 with the following stacktrace:
 {code}
 java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be 
 cast to org.apache.hadoop.io.LongWritable
   at 
 org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84)
   at 
 org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253)
   at 
 org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
   at 
 org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242)
   at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52)
   at 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
   at 
 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
   at 
 org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
   at 
 org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
 {code}
 The proximal cause of this failure is that the AvroContainerOutputFormat's 
 signature mandates a LongWritable key and HCat's FileRecordWriterContainer 
 forces a NullWritable. I'm not sure of a general fix, other than redefining 
 HiveOutputFormat to mandate a WritableComparable.
 It looks like accepting WritableComparable is what's done in the other Hive 
 OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also 
 be changed, since it's ignoring the key. That way fixing things so 
 FileRecordWriterContainer can always use NullWritable could get spun into a 
 different issue?
 The underlying cause for failure to write to AvroSerde tables is that 
 AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so 
 fixing the above will just push the failure into the placeholder RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HIVE-6794) select * from parquet hive table containing map columns runs into exception


 [ 
https://issues.apache.org/jira/browse/HIVE-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho resolved HIVE-6794.
-

Resolution: Fixed

This is already solved by HIVE-6575, can you try with trunk?

 select * from parquet hive table containing map columns runs into exception
 ---

 Key: HIVE-6794
 URL: https://issues.apache.org/jira/browse/HIVE-6794
 Project: Hive
  Issue Type: Bug
  Components: File Formats, Serializers/Deserializers
Affects Versions: 0.13.0
Reporter: Tongjie Chen

 select * from parquet hive table containing map columns run into exception 
 FAILED: RuntimeException java.lang.ClassCastException: 
 parquet.hive.serde.DeepParquetHiveMapInspector cannot be cast to 
 parquet.hive.serde.StandardParquetHiveMapInspector
 To reproduce, use the following steps (both regular_map_table and 
 parquet_map_table contains schema c1 , c2 int):
 hive insert overwrite table parquet_map_table select * from 
 regular_map_table;
 hive select * from parquet_map_table; (the first query to select * works 
 fine )
 OK
 {key1:value1} 1
 {key2:value2} 2
 {key3:value3} 3
 Time taken: 2.669 seconds, Fetched: 3 row(s)
 hive select * from parquet_map_table; (the second and all subsequent exact 
 same query breaks)
 FAILED: RuntimeException java.lang.ClassCastException: 
 parquet.hive.serde.DeepParquetHiveMapInspector cannot be cast to 
 parquet.hive.serde.StandardParquetHiveMapInspector
 Interestingly select c1 from parquet_map_column query works fine though.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6792) hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS

2014-03-31 Thread Chaoyu Tang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-6792:
--

Attachment: HIVE-6792-1.patch

Thanks, Szehon, for pointing out. I changed to use member variable conf instead 
and please see attached HIVE-6792-1.patch

 hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS
 --

 Key: HIVE-6792
 URL: https://issues.apache.org/jira/browse/HIVE-6792
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Security
Affects Versions: 0.14.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
 Attachments: HIVE-6792-1.patch, HIVE-6792.patch


 hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS. When it 
 is set to true, the table created using create table .. as select.. does not 
 inherit its parent directory's group and permission mode. It can be easily 
 reproduced:
 ==
 hive dfs -ls -R /user/hive/warehouse;
 drwxrwx--T   - hive   hive0 2014-03-30 17:44 
 /user/hive/warehouse/ctas.db
 drwxr-xr-x   - hive   hive0 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl
 -rw-r--r--   3 hive   hive46059 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl/00_0
 hive create table ctas.test_perm as select * from ctas_src_tbl;
 
 hive dfs -ls -R /user/hive/warehouse;  
 drwxrwx--T   - hive   hive0 2014-03-30 17:46 
 /user/hive/warehouse/ctas.db
 drwxr-xr-x   - hive   supergroup  0 2014-03-30 17:46 
 /user/hive/warehouse/ctas.db/test_perm
 -rw-r--r--   3 hive   supergroup  46059 2014-03-30 17:46 
 /user/hive/warehouse/ctas.db/test_perm/00_0
 drwxr-xr-x   - hive   hive0 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl
 -rw-r--r--   3 hive   hive46059 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl/00_0
 ==
 The created table does not inherit its database ctas's group hive and 
 permission mode 770, instead it takes the default group (supergroup) and 
 permission mode (755) in hdfs



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6792) hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS

2014-03-31 Thread Chaoyu Tang (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaoyu Tang updated HIVE-6792:
--

Status: Patch Available  (was: Open)

 hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS
 --

 Key: HIVE-6792
 URL: https://issues.apache.org/jira/browse/HIVE-6792
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Security
Affects Versions: 0.14.0
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang
 Attachments: HIVE-6792-1.patch, HIVE-6792.patch


 hive.warehouse.subdir.inherit.perms doesn't work correctly in CTAS. When it 
 is set to true, the table created using create table .. as select.. does not 
 inherit its parent directory's group and permission mode. It can be easily 
 reproduced:
 ==
 hive dfs -ls -R /user/hive/warehouse;
 drwxrwx--T   - hive   hive0 2014-03-30 17:44 
 /user/hive/warehouse/ctas.db
 drwxr-xr-x   - hive   hive0 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl
 -rw-r--r--   3 hive   hive46059 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl/00_0
 hive create table ctas.test_perm as select * from ctas_src_tbl;
 
 hive dfs -ls -R /user/hive/warehouse;  
 drwxrwx--T   - hive   hive0 2014-03-30 17:46 
 /user/hive/warehouse/ctas.db
 drwxr-xr-x   - hive   supergroup  0 2014-03-30 17:46 
 /user/hive/warehouse/ctas.db/test_perm
 -rw-r--r--   3 hive   supergroup  46059 2014-03-30 17:46 
 /user/hive/warehouse/ctas.db/test_perm/00_0
 drwxr-xr-x   - hive   hive0 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl
 -rw-r--r--   3 hive   hive46059 2014-03-30 17:20 
 /user/hive/warehouse/ctas_src_tbl/00_0
 ==
 The created table does not inherit its database ctas's group hive and 
 permission mode 770, instead it takes the default group (supergroup) and 
 permission mode (755) in hdfs



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6374) Hive job submitted with non-default name node (fs.default.name) doesn't process locations properly


 [ 
https://issues.apache.org/jira/browse/HIVE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6374:
---

Assignee: Benjamin Zhitomirsky

 Hive job submitted with non-default name node (fs.default.name) doesn't 
 process locations properly 
 ---

 Key: HIVE-6374
 URL: https://issues.apache.org/jira/browse/HIVE-6374
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 0.11.0, 0.12.0, 0.13.0
 Environment: Any
Reporter: Benjamin Zhitomirsky
Assignee: Benjamin Zhitomirsky
 Attachments: Design of the fix HIVE-6374.docx, hive-6374.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 Create table/index/database and add partition DDL doesn't work properly if 
 all following conditions are true:
 - Metastore service is used
 - fs.default.name is specified and it differs from the default one
 - Location is not specified or specified as a not fully qualified URI
 The root cause of this behavior is that Hive client doesn't pass 
 configuration context to the metastore services which tries to resolve the 
 paths. The fix is it too resolve the path in the Hive client if 
 fs.default.name is specified and it differs from the default one (it is must 
 easier then start passing the context, which would be a major change).
 The CR will submitted shortly after tests are done



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6374) Hive job submitted with non-default name node (fs.default.name) doesn't process locations properly


[ 
https://issues.apache.org/jira/browse/HIVE-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955752#comment-13955752
 ] 

Ashutosh Chauhan commented on HIVE-6374:


[~benjzh] Can you create review board entry for this on : 
https://reviews.apache.org/r/new/

 Hive job submitted with non-default name node (fs.default.name) doesn't 
 process locations properly 
 ---

 Key: HIVE-6374
 URL: https://issues.apache.org/jira/browse/HIVE-6374
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 0.11.0, 0.12.0, 0.13.0
 Environment: Any
Reporter: Benjamin Zhitomirsky
Assignee: Benjamin Zhitomirsky
 Attachments: Design of the fix HIVE-6374.docx, hive-6374.patch

   Original Estimate: 168h
  Remaining Estimate: 168h

 Create table/index/database and add partition DDL doesn't work properly if 
 all following conditions are true:
 - Metastore service is used
 - fs.default.name is specified and it differs from the default one
 - Location is not specified or specified as a not fully qualified URI
 The root cause of this behavior is that Hive client doesn't pass 
 configuration context to the metastore services which tries to resolve the 
 paths. The fix is it too resolve the path in the Hive client if 
 fs.default.name is specified and it differs from the default one (it is must 
 easier then start passing the context, which would be a major change).
 The CR will submitted shortly after tests are done



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pala M Muthaia updated HIVE-6131:
-

Attachment: (was: HIVE-6131.1.patch.txt)

New columns after table alter result in null values despite data

Key: HIVE-6131
URL: https://issues.apache.org/jira/browse/HIVE-6131
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: James Vaughan
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pala M Muthaia updated HIVE-6131:
-

Attachment: HIVE-6131.1.patch

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955767#comment-13955767
]

Pala M Muthaia commented on HIVE-6131:
--

[~szehon], i have reuploaded the patch with expected name. I still don't see a
job in progress for jenkins Hive precommit build. Let me know if something else
is needed.

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955772#comment-13955772
]

Szehon Ho commented on HIVE-6131:
-

It should be there, are you looking at
[http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/|http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/]?
I think its either build 2054-2056 depending on when it was uploaded. Let's
wait for those and see.

New columns after table alter result in null values despite data

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6795) metastore initialization should add default roles with default, SBA


 [ 
https://issues.apache.org/jira/browse/HIVE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6795:


Affects Version/s: 0.13.0

 metastore initialization should add default roles with default, SBA
 ---

 Key: HIVE-6795
 URL: https://issues.apache.org/jira/browse/HIVE-6795
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-6795.1.patch


 Hiveserver2 running sql standard authorization can connect to a metastore 
 running storage based authorization. Currently metastore is not adding the 
 standard roles to the db in such cases.
 It would be better to add them in these cases as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6795) metastore initialization should add default roles with default, SBA


 [ 
https://issues.apache.org/jira/browse/HIVE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6795:


Attachment: HIVE-6795.1.patch

 metastore initialization should add default roles with default, SBA
 ---

 Key: HIVE-6795
 URL: https://issues.apache.org/jira/browse/HIVE-6795
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-6795.1.patch


 Hiveserver2 running sql standard authorization can connect to a metastore 
 running storage based authorization. Currently metastore is not adding the 
 standard roles to the db in such cases.
 It would be better to add them in these cases as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6795) metastore initialization should add default roles with default, SBA


 [ 
https://issues.apache.org/jira/browse/HIVE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6795:


Status: Patch Available  (was: Open)

 metastore initialization should add default roles with default, SBA
 ---

 Key: HIVE-6795
 URL: https://issues.apache.org/jira/browse/HIVE-6795
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-6795.1.patch


 Hiveserver2 running sql standard authorization can connect to a metastore 
 running storage based authorization. Currently metastore is not adding the 
 standard roles to the db in such cases.
 It would be better to add them in these cases as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6783) Incompatible schema for maps between parquet-hive and parquet-pig


[ 
https://issues.apache.org/jira/browse/HIVE-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955782#comment-13955782
 ] 

Szehon Ho commented on HIVE-6783:
-

Thanks, that is my only concern to not break existing stored maps, I'm ok if 
thats the case.  Although I wonder, it would seem pig should be able to read 
hive's maps if that is the case, because we are using pig's schema now to read? 
 Or is there some difference there?


 Incompatible schema for maps between parquet-hive and parquet-pig
 -

 Key: HIVE-6783
 URL: https://issues.apache.org/jira/browse/HIVE-6783
 Project: Hive
  Issue Type: Bug
  Components: File Formats
Affects Versions: 0.13.0
Reporter: Tongjie Chen
 Fix For: 0.13.0

 Attachments: HIVE-6783.1.patch.txt, HIVE-6783.2.patch.txt, 
 HIVE-6783.3.patch.txt, HIVE-6783.4.patch.txt


 see also in following parquet issue:
 https://github.com/Parquet/parquet-mr/issues/290
 The schema written for maps isn't compatible between hive and pig. This means 
 any files written in one cannot be properly read in the other.
 More specifically,  for the same map column c1, parquet-pig generates schema:
 message pig_schema {
   optional group c1 (MAP) {
 repeated group map (MAP_KEY_VALUE) {
   required binary key (UTF8);
   optional binary value;
 }   
   }
 }
 while parquet-hive generates schema:
 message hive_schema {
optional group c1 (MAP_KEY_VALUE) {
  repeated group map {
required binary key;
optional binary value;
}
  }
 }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6795) metastore initialization should add default roles with default, SBA


[ 
https://issues.apache.org/jira/browse/HIVE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955787#comment-13955787
 ] 

Ashutosh Chauhan commented on HIVE-6795:


+1

 metastore initialization should add default roles with default, SBA
 ---

 Key: HIVE-6795
 URL: https://issues.apache.org/jira/browse/HIVE-6795
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-6795.1.patch


 Hiveserver2 running sql standard authorization can connect to a metastore 
 running storage based authorization. Currently metastore is not adding the 
 standard roles to the db in such cases.
 It would be better to add them in these cases as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-2818) Create table should check privilege of target database, not default database


 [ 
https://issues.apache.org/jira/browse/HIVE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-2818:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

This doesn't seem to be a problem any longer. Feel free to reopen if thats not 
the case.

 Create table should check privilege of target database, not default database
 

 Key: HIVE-2818
 URL: https://issues.apache.org/jira/browse/HIVE-2818
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Security
Affects Versions: 0.7.1
Reporter: Benyi Wang
Assignee: Navis
 Fix For: 0.13.0

 Attachments: HIVE-2818.1.patch.txt, HIVE-2818.2.patch.txt, 
 HIVE-2818.3.patch.txt, HIVE-2818.4.patch.txt, HIVE-2818.5.patch.txt, 
 HIVE-2818.6.patch.txt


 Hive seems check the current database to determine the privilege of a 
 statement when you use fully qualified name like 'database.table'
  
 {code}
 hive set hive.security.authorization.enabled=true;
 hive create database test_db;
 hive grant all on database test_db to user test_user;
 hive revoke all on database default from test_user;
 hive use default;
 hive create table test_db.new_table (id int);
 Authorization failed:No privilege 'Create' found for outputs { 
 database:default}. Use show grant to get more details.
 hive use test_db;
 hive create table test_db.new_table (id int);
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6795) metastore initialization should add default roles with default, SBA


 [ 
https://issues.apache.org/jira/browse/HIVE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6795:


Reporter: Deepesh Khandelwal  (was: Thejas M Nair)

 metastore initialization should add default roles with default, SBA
 ---

 Key: HIVE-6795
 URL: https://issues.apache.org/jira/browse/HIVE-6795
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 0.13.0
Reporter: Deepesh Khandelwal
Assignee: Thejas M Nair
 Attachments: HIVE-6795.1.patch


 Hiveserver2 running sql standard authorization can connect to a metastore 
 running storage based authorization. Currently metastore is not adding the 
 standard roles to the db in such cases.
 It would be better to add them in these cases as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6796) Create/drop roles is case-sensitive whereas 'set role' is case insensitive

Thejas M Nair created HIVE-6796:
---

 Summary: Create/drop roles is case-sensitive whereas 'set role' is 
case insensitive
 Key: HIVE-6796
 URL: https://issues.apache.org/jira/browse/HIVE-6796
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair


Create/drop role operations should be case insensitive.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-6796) Create/drop roles is case-sensitive whereas 'set role' is case insensitive


 [ 
https://issues.apache.org/jira/browse/HIVE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair reassigned HIVE-6796:
---

Assignee: Thejas M Nair

 Create/drop roles is case-sensitive whereas 'set role' is case insensitive
 --

 Key: HIVE-6796
 URL: https://issues.apache.org/jira/browse/HIVE-6796
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair
Assignee: Thejas M Nair

 Create/drop role operations should be case insensitive.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6796) Create/drop roles is case-sensitive whereas 'set role' is case insensitive


 [ 
https://issues.apache.org/jira/browse/HIVE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6796:


Reporter: Deepesh Khandelwal  (was: Thejas M Nair)

 Create/drop roles is case-sensitive whereas 'set role' is case insensitive
 --

 Key: HIVE-6796
 URL: https://issues.apache.org/jira/browse/HIVE-6796
 Project: Hive
  Issue Type: Bug
Reporter: Deepesh Khandelwal
Assignee: Thejas M Nair

 Create/drop role operations should be case insensitive.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6796) Create/drop roles is case-sensitive whereas 'set role' is case insensitive


 [ 
https://issues.apache.org/jira/browse/HIVE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6796:


Assignee: Ashutosh Chauhan  (was: Thejas M Nair)

 Create/drop roles is case-sensitive whereas 'set role' is case insensitive
 --

 Key: HIVE-6796
 URL: https://issues.apache.org/jira/browse/HIVE-6796
 Project: Hive
  Issue Type: Bug
Reporter: Deepesh Khandelwal
Assignee: Ashutosh Chauhan

 Create/drop role operations should be case insensitive.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6796) Create/drop roles is case-sensitive whereas 'set role' is case insensitive


[ 
https://issues.apache.org/jira/browse/HIVE-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955792#comment-13955792
 ] 

Thejas M Nair commented on HIVE-6796:
-

From [~deepesh]

Steps to reproduce:
# Login as admin user (eg hrt_qa) and create a new role.
{noformat}
0: jdbc:hive2://hor8n19.gq1.ygridcore.net:100 set role ADMIN;
No rows affected (0.048 seconds)
0: jdbc:hive2://:10 create role TESTROLE;
No rows affected (0.081 seconds)
0: jdbc:hive2://:10 grant role TESTROLE to user hrt_1;
No rows affected (0.086 seconds)
0: jdbc:hive2://:10 show roles;
+---+
|   role|
+---+
| ADMIN |
| PUBLIC|
| TESTROLE  |
|   |
+---+
4 rows selected (0.05 seconds)
{noformat}
# Login as public user hrt_1 and assume above role.
{noformat}
beeline !connect jdbc:hive2://:1 hrt_1 pwd
Connected to: Apache Hive (version 0.13.0.2.1.1.0-261)
Driver: Hive JDBC (version 0.13.0.2.1.1.0-261)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://:10 show current roles;
+---+
|   role|
+---+
| PUBLIC|
| TESTROLE  |
|   |
+---+
3 rows selected (0.096 seconds)
0: jdbc:hive2://:10 set role testrole;
No rows affected (0.025 seconds)
0: jdbc:hive2://:10 show current roles;
+---+
|   role|
+---+
| TESTROLE  |
|   |
+---+
2 rows selected (0.019 seconds)
{noformat}
This seems convenient as to not worry about case-sensitivity.
# But here is the problem. Try to create role testrole (earlier we created 
TESTROLE). On admin session (hrt_qa):
{noformat}
0: jdbc:hive2://:10 create role testrole;
No rows affected (0.15 seconds)
0: jdbc:hive2://:10 show roles;
+---+
|   role|
+---+
| ADMIN |
| PUBLIC|
| TESTROLE  |
| testrole  |
|   |
+---+
5 rows selected (0.026 seconds)
0: jdbc:hive2://:10 drop role TESTROLE;
No rows affected (0.094 seconds)
0: jdbc:hive2://:10 show roles;
+---+
|   role|
+---+
| ADMIN |
| PUBLIC|
| testrole  |
|   |
+---+
4 rows selected (0.026 seconds)
{noformat}

Above shows that create/drop role is case sensitive but set role is not. They 
should be all consistent (either they all are case sensitive or they are all 
case insensitive).

 Create/drop roles is case-sensitive whereas 'set role' is case insensitive
 --

 Key: HIVE-6796
 URL: https://issues.apache.org/jira/browse/HIVE-6796
 Project: Hive
  Issue Type: Bug
Reporter: Deepesh Khandelwal
Assignee: Thejas M Nair

 Create/drop role operations should be case insensitive.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6789) HiveStatement client transport lock should unlock in finally block.


[ 
https://issues.apache.org/jira/browse/HIVE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955819#comment-13955819
 ] 

Thejas M Nair commented on HIVE-6789:
-

+1

 HiveStatement client transport lock should unlock in finally block.
 ---

 Key: HIVE-6789
 URL: https://issues.apache.org/jira/browse/HIVE-6789
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6789.1.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6779) Hive cli may get into inconsistent state when Ctrl-C is hit on hadoop2


 [ 
https://issues.apache.org/jira/browse/HIVE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6779:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk  0.13

 Hive cli may get into inconsistent state when Ctrl-C is hit on hadoop2
 --

 Key: HIVE-6779
 URL: https://issues.apache.org/jira/browse/HIVE-6779
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
 Fix For: 0.13.0

 Attachments: HIVE-6779.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory

2014-03-31 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955836#comment-13955836
 ] 

Hive QA commented on HIVE-6793:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12637896/HIVE-6793.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5513 tests executed
*Failed tests:*
{noformat}
org.apache.hive.service.cli.thrift.TestThriftHttpCLIService.testExecuteStatementAsync
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2052/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2052/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12637896

 DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
 --

 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Query Processor
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Attachments: HIVE-6793.patch


 Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
 HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
 related task creations such as for analyzeShowRolePrincipals(). This JIRA is 
 to make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory


[ 
https://issues.apache.org/jira/browse/HIVE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955841#comment-13955841
 ] 

Xuefu Zhang commented on HIVE-6793:
---

the above test failure isn't related to the patch. It also appears in other 
test runs.

 DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
 --

 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Query Processor
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Attachments: HIVE-6793.patch


 Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
 HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
 related task creations such as for analyzeShowRolePrincipals(). This JIRA is 
 to make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6783) Incompatible schema for maps between parquet-hive and parquet-pig

2014-03-31 Thread Tongjie Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955845#comment-13955845
 ] 

Tongjie Chen commented on HIVE-6783:


The fix presented in this jira is to tag map type with the appropriate 
OriginalType.

The reason hive is backward compatible is that when parquet-hive converts 
parquet fields map.key and map.value back to hive map column, it does not check 
MAP_KEY_VALUE OriginalType.  Also, the equals method of GroupType does not 
check OriginalType at all, hence hive_schema and pig_schema shown in 
description section are treated as equal (the only different is OriginalType).

However, parquet-pig's PigSchemaConverter checks to make sure map's 
OriginalType is of correct type, hence it breaks when it reads Hive's map.

With the fix, pig now can read hive's map since hive converts it right.


 Incompatible schema for maps between parquet-hive and parquet-pig
 -

 Key: HIVE-6783
 URL: https://issues.apache.org/jira/browse/HIVE-6783
 Project: Hive
  Issue Type: Bug
  Components: File Formats
Affects Versions: 0.13.0
Reporter: Tongjie Chen
 Fix For: 0.13.0

 Attachments: HIVE-6783.1.patch.txt, HIVE-6783.2.patch.txt, 
 HIVE-6783.3.patch.txt, HIVE-6783.4.patch.txt


 see also in following parquet issue:
 https://github.com/Parquet/parquet-mr/issues/290
 The schema written for maps isn't compatible between hive and pig. This means 
 any files written in one cannot be properly read in the other.
 More specifically,  for the same map column c1, parquet-pig generates schema:
 message pig_schema {
   optional group c1 (MAP) {
 repeated group map (MAP_KEY_VALUE) {
   required binary key (UTF8);
   optional binary value;
 }   
   }
 }
 while parquet-hive generates schema:
 message hive_schema {
optional group c1 (MAP_KEY_VALUE) {
  repeated group map {
required binary key;
optional binary value;
}
  }
 }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6793) DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory


[ 
https://issues.apache.org/jira/browse/HIVE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955847#comment-13955847
 ] 

Ashutosh Chauhan commented on HIVE-6793:


+1

 DDLSemanticAnalyzer.analyzeShowRoles() should use HiveAuthorizationTaskFactory
 --

 Key: HIVE-6793
 URL: https://issues.apache.org/jira/browse/HIVE-6793
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Query Processor
Affects Versions: 0.13.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Attachments: HIVE-6793.patch


 Currently DDLSemanticAnalyzer.analyzeShowRoles() isn't using 
 HiveAuthorizationTaskFactory to create task, at odds with other Authorization 
 related task creations such as for analyzeShowRolePrincipals(). This JIRA is 
 to make it consistent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6783) Incompatible schema for maps between parquet-hive and parquet-pig


[ 
https://issues.apache.org/jira/browse/HIVE-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955851#comment-13955851
 ] 

Szehon Ho commented on HIVE-6783:
-

Thanks looks good to me.  + [~brocknoland], [~xuefuz]

 Incompatible schema for maps between parquet-hive and parquet-pig
 -

 Key: HIVE-6783
 URL: https://issues.apache.org/jira/browse/HIVE-6783
 Project: Hive
  Issue Type: Bug
  Components: File Formats
Affects Versions: 0.13.0
Reporter: Tongjie Chen
 Fix For: 0.13.0

 Attachments: HIVE-6783.1.patch.txt, HIVE-6783.2.patch.txt, 
 HIVE-6783.3.patch.txt, HIVE-6783.4.patch.txt


 see also in following parquet issue:
 https://github.com/Parquet/parquet-mr/issues/290
 The schema written for maps isn't compatible between hive and pig. This means 
 any files written in one cannot be properly read in the other.
 More specifically,  for the same map column c1, parquet-pig generates schema:
 message pig_schema {
   optional group c1 (MAP) {
 repeated group map (MAP_KEY_VALUE) {
   required binary key (UTF8);
   optional binary value;
 }   
   }
 }
 while parquet-hive generates schema:
 message hive_schema {
optional group c1 (MAP_KEY_VALUE) {
  repeated group map {
required binary key;
optional binary value;
}
  }
 }



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6797) Add protection against divide by zero in stats annotation

Prasanth J created HIVE-6797:


 Summary: Add protection against divide by zero in stats annotation
 Key: HIVE-6797
 URL: https://issues.apache.org/jira/browse/HIVE-6797
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
 Fix For: 0.13.0


In stats annotation, the denominator computation in join operator is not 
protected for divide by zero exception. It will be an issue when NDV (count 
distinct) updated by updateStats() becomes 0. This patch adds protection in 
updateStats() method to prevent divide-by-zero in downstream operators.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6798) Update column stats based on filter expression in stats annotation

Prasanth J created HIVE-6798:


 Summary: Update column stats based on filter expression in stats 
annotation
 Key: HIVE-6798
 URL: https://issues.apache.org/jira/browse/HIVE-6798
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6797) Add protection against divide by zero in stats annotation


 [ 
https://issues.apache.org/jira/browse/HIVE-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6797:
-

Attachment: HIVE-6797.1.patch

 Add protection against divide by zero in stats annotation
 -

 Key: HIVE-6797
 URL: https://issues.apache.org/jira/browse/HIVE-6797
 Project: Hive
  Issue Type: Sub-task
  Components: Query Processor, Statistics
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
 Fix For: 0.13.0

 Attachments: HIVE-6797.1.patch


 In stats annotation, the denominator computation in join operator is not 
 protected for divide by zero exception. It will be an issue when NDV (count 
 distinct) updated by updateStats() becomes 0. This patch adds protection in 
 updateStats() method to prevent divide-by-zero in downstream operators.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6797) Add protection against divide by zero in stats annotation


 [ 
https://issues.apache.org/jira/browse/HIVE-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6797:
-

Status: Patch Available  (was: Open)

 Add protection against divide by zero in stats annotation
 -

 Key: HIVE-6797
 URL: https://issues.apache.org/jira/browse/HIVE-6797
 Project: Hive
  Issue Type: Sub-task
  Components: Query Processor, Statistics
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
 Fix For: 0.13.0

 Attachments: HIVE-6797.1.patch


 In stats annotation, the denominator computation in join operator is not 
 protected for divide by zero exception. It will be an issue when NDV (count 
 distinct) updated by updateStats() becomes 0. This patch adds protection in 
 updateStats() method to prevent divide-by-zero in downstream operators.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site

Vaibhav Gumashta created HIVE-6800:
--

 Summary: HiveServer2 is not passing proxy user setting through 
hive-site
 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0


Setting the following in core-site.xml works fine in a secure cluster with 
hive.server2.allow.user.substitution set to true:
{code}
property
  namehadoop.proxyuser.user1.groups/name
  valueusers/value
/property

property
  namehadoop.proxyuser.user1.hosts/name
  value*/value
/property
{code}

where user1 will be proxying for user2:
{code}
!connect 
jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
 user1 fakepwd org.apache.hive.jdbc.HiveDriver
{code}

However, setting this in hive-site.xml throws Failed to validate proxy 
privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check

2014-03-31 Thread Dilli Arumugam (JIRA)

Dilli Arumugam created HIVE-6799:


 Summary: HiveServer2 needs to map kerberos name to local name 
before proxy check
 Key: HIVE-6799
 URL: https://issues.apache.org/jira/browse/HIVE-6799
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Dilli Arumugam


HiveServer2 does not map kerberos name of authenticated principal to local name.

Due to this, I get error like the following in HiveServer log:
Failed to validate proxy privilage of knox/hdps.example.com for sam

I have KINITED as knox/hdps.example@example.com

I do have the following in core-site.xml

  property
namehadoop.proxyuser.knox.groups/name
valueusers/value
  /property
  property
namehadoop.proxyuser.knox.hosts/name
value*/value
  /property

Other hadoop ecosystem services do this and call it auth to local mapping. 
This needs to be added to HiveServer2 as well.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site


 [ 
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6800:
---

Attachment: HIVE-6800.1.patch

cc [~thejas] [~prasadm]

[~rhbutani] Bug for 13!

Thanks!

 HiveServer2 is not passing proxy user setting through hive-site
 ---

 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6800.1.patch


 Setting the following in core-site.xml works fine in a secure cluster with 
 hive.server2.allow.user.substitution set to true:
 {code}
 property
   namehadoop.proxyuser.user1.groups/name
   valueusers/value
 /property
 
 property
   namehadoop.proxyuser.user1.hosts/name
   value*/value
 /property
 {code}
 where user1 will be proxying for user2:
 {code}
 !connect 
 jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
  user1 fakepwd org.apache.hive.jdbc.HiveDriver
 {code}
 However, setting this in hive-site.xml throws Failed to validate proxy 
 privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6801) beeline kerberos authentication fails if the client principal name has hostname part

2014-03-31 Thread Dilli Arumugam (JIRA)

Dilli Arumugam created HIVE-6801:


 Summary: beeline kerberos authentication fails if the client 
principal name has hostname part
 Key: HIVE-6801
 URL: https://issues.apache.org/jira/browse/HIVE-6801
 Project: Hive
  Issue Type: Improvement
  Components: Clients
Reporter: Dilli Arumugam
Assignee: Dilli Arumugam


Kinited as gu...@example.com

Connected successfully with beeline using command
!connect 
jdbc:hive2://hdps.example.com:10001/default;principal=hive/hdps.example@example.com;hive.server2.proxy.user=guest?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice
  dummy dummy-pass org.apache.hive.jdbc.HiveDriver

Kinited as bob/hdps.exmaple@example.com

!connect 
jdbc:hive2://hdps.example.com:10001/default;principal=hive/hdps.example@example.com;hive.server2.proxy.user=guest?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice
  dummy dummy-pass org.apache.hive.jdbc.HiveDriver

Failed with stack trace
Error: Could not establish connection to 
jdbc:hive2://hdps.example.com:10001/default;principal=hive/hdps.example@example.com;hive.server2.proxy.user=guest?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice:
 org.apache.http.client.ClientProtocolException (state=08S01,code=0)
java.sql.SQLException: Could not establish connection to 
jdbc:hive2://hdps.example.com:10001/default;principal=hive/hdps.example@example.com;hive.server2.proxy.user=guest?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice:
 org.apache.http.client.ClientProtocolException
at 
org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:426)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:193)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:154)
at 
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at 
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:186)
at org.apache.hive.beeline.Commands.connect(Commands.java:959)
at org.apache.hive.beeline.Commands.connect(Commands.java:880)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:44)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:792)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:659)
at 
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:368)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:351)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.thrift.transport.TTransportException: 
org.apache.http.client.ClientProtocolException
at 
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:281)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:297)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
at 
org.apache.hive.service.cli.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:150)
at 
org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:142)
at 
org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:415)
... 22 more
Caused by: org.apache.http.client.ClientProtocolException
at 
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
at 
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:827)
at 
org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:235)
... 27 more
Caused by: org.apache.http.HttpException
at 
org.apache.hive.jdbc.HttpKerberosRequestInterceptor.process(HttpKerberosRequestInterceptor.java:67)
at 
org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:109)
at 
org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:176)
at

[jira] [Assigned] (HIVE-6799) HiveServer2 needs to map kerberos name to local name before proxy check

2014-03-31 Thread Dilli Arumugam (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dilli Arumugam reassigned HIVE-6799:


Assignee: Dilli Arumugam

 HiveServer2 needs to map kerberos name to local name before proxy check
 ---

 Key: HIVE-6799
 URL: https://issues.apache.org/jira/browse/HIVE-6799
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Dilli Arumugam
Assignee: Dilli Arumugam

 HiveServer2 does not map kerberos name of authenticated principal to local 
 name.
 Due to this, I get error like the following in HiveServer log:
 Failed to validate proxy privilage of knox/hdps.example.com for sam
 I have KINITED as knox/hdps.example@example.com
 I do have the following in core-site.xml
   property
 namehadoop.proxyuser.knox.groups/name
 valueusers/value
   /property
   property
 namehadoop.proxyuser.knox.hosts/name
 value*/value
   /property
 Other hadoop ecosystem services do this and call it auth to local mapping. 
 This needs to be added to HiveServer2 as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site


 [ 
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6800:
---

Attachment: (was: HIVE-6800.1.patch)

 HiveServer2 is not passing proxy user setting through hive-site
 ---

 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6800.1.patch


 Setting the following in core-site.xml works fine in a secure cluster with 
 hive.server2.allow.user.substitution set to true:
 {code}
 property
   namehadoop.proxyuser.user1.groups/name
   valueusers/value
 /property
 
 property
   namehadoop.proxyuser.user1.hosts/name
   value*/value
 /property
 {code}
 where user1 will be proxying for user2:
 {code}
 !connect 
 jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
  user1 fakepwd org.apache.hive.jdbc.HiveDriver
 {code}
 However, setting this in hive-site.xml throws Failed to validate proxy 
 privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site


 [ 
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6800:
---

Attachment: HIVE-6800.1.patch

 HiveServer2 is not passing proxy user setting through hive-site
 ---

 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6800.1.patch


 Setting the following in core-site.xml works fine in a secure cluster with 
 hive.server2.allow.user.substitution set to true:
 {code}
 property
   namehadoop.proxyuser.user1.groups/name
   valueusers/value
 /property
 
 property
   namehadoop.proxyuser.user1.hosts/name
   value*/value
 /property
 {code}
 where user1 will be proxying for user2:
 {code}
 !connect 
 jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
  user1 fakepwd org.apache.hive.jdbc.HiveDriver
 {code}
 However, setting this in hive-site.xml throws Failed to validate proxy 
 privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site


 [ 
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6800:
---

Status: Patch Available  (was: Open)

 HiveServer2 is not passing proxy user setting through hive-site
 ---

 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6800.1.patch


 Setting the following in core-site.xml works fine in a secure cluster with 
 hive.server2.allow.user.substitution set to true:
 {code}
 property
   namehadoop.proxyuser.user1.groups/name
   valueusers/value
 /property
 
 property
   namehadoop.proxyuser.user1.hosts/name
   value*/value
 /property
 {code}
 where user1 will be proxying for user2:
 {code}
 !connect 
 jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
  user1 fakepwd org.apache.hive.jdbc.HiveDriver
 {code}
 However, setting this in hive-site.xml throws Failed to validate proxy 
 privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6739) Hive HBase query fails on Tez due to missing jars and then due to NPE in getSplits


 [ 
https://issues.apache.org/jira/browse/HIVE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6739:
---

Attachment: HIVE-6739.01.patch

Updated patch. Note that this depends on a feature in yet-unreleased Tez 0.4; 
if that ships first I will update pom, otherwise it will be split into two 
patches. We will commit restart-AM path here, and relocalize path separately in 
Hive 0.14

 Hive HBase query fails on Tez due to missing jars and then due to NPE in 
 getSplits
 --

 Key: HIVE-6739
 URL: https://issues.apache.org/jira/browse/HIVE-6739
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Fix For: 0.13.0

 Attachments: HIVE-6739.01.patch, HIVE-6739.patch, 
 HIVE-6739.preliminary.patch


 Tez paths in Hive never call configure on the input/output operators, so 
 (among other things, potentially) requisite files never get added to the job



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site


 [ 
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6800:
---

Attachment: HIVE-6800.1.patch

 HiveServer2 is not passing proxy user setting through hive-site
 ---

 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6800.1.patch


 Setting the following in core-site.xml works fine in a secure cluster with 
 hive.server2.allow.user.substitution set to true:
 {code}
 property
   namehadoop.proxyuser.user1.groups/name
   valueusers/value
 /property
 
 property
   namehadoop.proxyuser.user1.hosts/name
   value*/value
 /property
 {code}
 where user1 will be proxying for user2:
 {code}
 !connect 
 jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
  user1 fakepwd org.apache.hive.jdbc.HiveDriver
 {code}
 However, setting this in hive-site.xml throws Failed to validate proxy 
 privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6800) HiveServer2 is not passing proxy user setting through hive-site


 [ 
https://issues.apache.org/jira/browse/HIVE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6800:
---

Attachment: (was: HIVE-6800.1.patch)

 HiveServer2 is not passing proxy user setting through hive-site
 ---

 Key: HIVE-6800
 URL: https://issues.apache.org/jira/browse/HIVE-6800
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0

 Attachments: HIVE-6800.1.patch


 Setting the following in core-site.xml works fine in a secure cluster with 
 hive.server2.allow.user.substitution set to true:
 {code}
 property
   namehadoop.proxyuser.user1.groups/name
   valueusers/value
 /property
 
 property
   namehadoop.proxyuser.user1.hosts/name
   value*/value
 /property
 {code}
 where user1 will be proxying for user2:
 {code}
 !connect 
 jdbc:hive2:/myhostname:1/;principal=hive/_h...@example.com;hive.server2.proxy.user=user2
  user1 fakepwd org.apache.hive.jdbc.HiveDriver
 {code}
 However, setting this in hive-site.xml throws Failed to validate proxy 
 privilage exception.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HIVE-6775) Altering partition column type should not be allowed when there is existing partitions for which column types cannot be casted to the new type.


 [ 
https://issues.apache.org/jira/browse/HIVE-6775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin resolved HIVE-6775.


Resolution: Duplicate

 Altering partition column type should not be allowed when there is existing 
 partitions for which column types cannot be casted to the new type.
 ---

 Key: HIVE-6775
 URL: https://issues.apache.org/jira/browse/HIVE-6775
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

2014-03-31 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955954#comment-13955954
 ] 

Hive QA commented on HIVE-6131:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12637927/HIVE-6131.1.patch

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 5514 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat14
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2054/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2054/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12637927

 New columns after table alter result in null values despite data
 

 Key: HIVE-6131
 URL: https://issues.apache.org/jira/browse/HIVE-6131
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: James Vaughan
Priority: Minor
 Attachments: HIVE-6131.1.patch


 Hi folks,
 I found and verified a bug on our CDH 4.0.3 install of Hive when adding 
 columns to tables with Partitions using 'REPLACE COLUMNS'.  I dug through the 
 Jira a little bit and didn't see anything for it so hopefully this isn't just 
 noise on the radar.
 Basically, when you alter a table with partitions and then reupload data to 
 that partition, it doesn't seem to recognize the extra data that actually 
 exists in HDFS- as in, returns NULL values on the new column despite having 
 the data and recognizing the new column in the metadata.
 Here's some steps to reproduce using a basic table:
 1.  Run this hive command:  CREATE TABLE jvaughan_test (col1 string) 
 partitioned by (day string);
 2.  Create a simple file on the system with a couple of entries, something 
 like hi and hi2 separated by newlines.
 3.  Run this hive command, pointing it at the file:  LOAD DATA LOCAL INPATH 
 'FILEDIR' OVERWRITE INTO TABLE jvaughan_test PARTITION (day = '2014-01-02');
 4.  Confirm the data with:  SELECT * FROM jvaughan_test WHERE day = 
 '2014-01-02';
 5.  Alter the column definitions:  ALTER TABLE jvaughan_test REPLACE COLUMNS 
 (col1 string, col2 string);
 6.  Edit your file and add a second column using the default separator 
 (ctrl+v, then ctrl+a in Vim) and add two more entries, such as hi3 on the 
 first row and hi4 on the second
 7.  Run step 3 again
 8.  Check the data again like in step 4
 For me, this is the results that get returned:
 hive select * from jvaughan_test where day = '2014-01-01';
 OK
 hiNULL2014-01-02
 hi2   NULL2014-01-02
 This is despite the fact that there is data in the file stored by the 
 partition in HDFS.
 Let me know if you need any other information.  The only workaround for me 
 currently is to drop partitions for any I'm replacing data in and THEN 
 reupload the new data file.
 Thanks,
 -James



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6802) Fix metastore.thrift: add partition_columns.types constant

Harish Butani created HIVE-6802:
---

 Summary: Fix metastore.thrift: add partition_columns.types constant
 Key: HIVE-6802
 URL: https://issues.apache.org/jira/browse/HIVE-6802
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani


HIVE-6642 edited the hive_metastoreConstants.java genned file. 
Need to add constant to thrift file and regen thrift classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6802) Fix metastore.thrift: add partition_columns.types constant


 [ 
https://issues.apache.org/jira/browse/HIVE-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6802:


Attachment: HIVE-6802.1.patch

 Fix metastore.thrift: add partition_columns.types constant
 --

 Key: HIVE-6802
 URL: https://issues.apache.org/jira/browse/HIVE-6802
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
 Attachments: HIVE-6802.1.patch


 HIVE-6642 edited the hive_metastoreConstants.java genned file. 
 Need to add constant to thrift file and regen thrift classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18179: Support more generic way of using composite key for HBaseHandler

2014-03-31 Thread Navis Ryu



 On March 25, 2014, 6:38 p.m., Xuefu Zhang wrote:
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseAbstractKeyFactory.java,
   line 31
  https://reviews.apache.org/r/18179/diff/6/?file=535090#file535090line31
 
  Do you think AbstractHBaseKeyFactory is slightly better?

Yes, it's conventionally better name. But I wanted related things adjacent to 
each other. You don't like it?


 On March 25, 2014, 6:38 p.m., Xuefu Zhang wrote:
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseDefaultKeyFactory.java,
   line 32
  https://reviews.apache.org/r/18179/diff/6/?file=535093#file535093line32
 
  DefaultHBaseKeyFactory?

Same with HBaseAbstractKeyFactory


 On March 25, 2014, 6:38 p.m., Xuefu Zhang wrote:
  serde/src/java/org/apache/hadoop/hive/serde2/StructObjectBaseInspector.java,
   line 28
  https://reviews.apache.org/r/18179/diff/6/?file=535121#file535121line28
 
  Does BaseStructObjectInspector sound better?

Same intention.


 On March 25, 2014, 6:38 p.m., Xuefu Zhang wrote:
  serde/src/java/org/apache/hadoop/hive/serde2/StructObjectBaseInspector.java,
   line 29
  https://reviews.apache.org/r/18179/diff/6/?file=535121#file535121line29
 
  I'm wondering what's the difference between this new class and the 
  existing LazySimpleStructObjectInspector class. Could we reuse rather than 
  creating new?

Wanted generic OI for all struct type regardless of SerDe but 
LazySimpleStructObjectInspector is felt like it's only for LazySimpleSerDe. I 
might rewrite LazySimpleStructObjectInspector with StructObjectBaseInspector.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18179/#review38465
---


On March 25, 2014, 8:02 a.m., Navis Ryu wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/18179/
 ---
 
 (Updated March 25, 2014, 8:02 a.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-6411
 https://issues.apache.org/jira/browse/HIVE-6411
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 HIVE-2599 introduced using custom object for the row key. But it forces key 
 objects to extend HBaseCompositeKey, which is again extension of LazyStruct. 
 If user provides proper Object and OI, we can replace internal key and keyOI 
 with those. 
 
 Initial implementation is based on factory interface.
 {code}
 public interface HBaseKeyFactory {
   void init(SerDeParameters parameters, Properties properties) throws 
 SerDeException;
   ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
   LazyObjectBase createObject(ObjectInspector inspector) throws 
 SerDeException;
 }
 {code}
 
 
 Diffs
 -
 
   hbase-handler/pom.xml 132af43 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java 
 PRE-CREATION 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseAbstractKeyFactory.java
  PRE-CREATION 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java 
 5008f15 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKeyFactory.java
  PRE-CREATION 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseDefaultKeyFactory.java
  PRE-CREATION 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseKeyFactory.java 
 PRE-CREATION 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java
  PRE-CREATION 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
 PRE-CREATION 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseScanRange.java 
 PRE-CREATION 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 
 b64590d 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
 4fe1b1b 
   
 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
  142bfd8 
   hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 
 fc40195 
   
 hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java
  13c344b 
   
 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java 
 PRE-CREATION 
   
 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory2.java 
 PRE-CREATION 
   
 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
 7c4fc9f 
   hbase-handler/src/test/queries/positive/hbase_custom_key.q PRE-CREATION 
   hbase-handler/src/test/queries/positive/hbase_custom_key2.q PRE-CREATION 
   hbase-handler/src/test/results/positive/hbase_custom_key.q.out PRE-CREATION 
   hbase-handler/src/test/results/positive/hbase_custom_key2.q.out 
 PRE-CREATION 
   itests/util/pom.xml e9720df

[jira] [Commented] (HIVE-6802) Fix metastore.thrift: add partition_columns.types constant


[ 
https://issues.apache.org/jira/browse/HIVE-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955963#comment-13955963
 ] 

Sergey Shelukhin commented on HIVE-6802:


+1

 Fix metastore.thrift: add partition_columns.types constant
 --

 Key: HIVE-6802
 URL: https://issues.apache.org/jira/browse/HIVE-6802
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
 Attachments: HIVE-6802.1.patch


 HIVE-6642 edited the hive_metastoreConstants.java genned file. 
 Need to add constant to thrift file and regen thrift classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6802) Fix metastore.thrift: add partition_columns.types constant

2014-03-31 Thread Jason Dere (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955961#comment-13955961
 ] 

Jason Dere commented on HIVE-6802:
--

+1

 Fix metastore.thrift: add partition_columns.types constant
 --

 Key: HIVE-6802
 URL: https://issues.apache.org/jira/browse/HIVE-6802
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
 Attachments: HIVE-6802.1.patch


 HIVE-6642 edited the hive_metastoreConstants.java genned file. 
 Need to add constant to thrift file and regen thrift classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18179: Support more generic way of using composite key for HBaseHandler

2014-03-31 Thread Navis Ryu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18179/
---

(Updated April 1, 2014, 12:59 a.m.)


Review request for hive.


Changes
---

Addressed some comments


Bugs: HIVE-6411
https://issues.apache.org/jira/browse/HIVE-6411


Repository: hive-git


Description
---

HIVE-2599 introduced using custom object for the row key. But it forces key 
objects to extend HBaseCompositeKey, which is again extension of LazyStruct. If 
user provides proper Object and OI, we can replace internal key and keyOI with 
those. 

Initial implementation is based on factory interface.
{code}
public interface HBaseKeyFactory {
  void init(SerDeParameters parameters, Properties properties) throws 
SerDeException;
  ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
  LazyObjectBase createObject(ObjectInspector inspector) throws SerDeException;
}
{code}


Diffs (updated)
-

  hbase-handler/pom.xml 132af43 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/ColumnMappings.java 
PRE-CREATION 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseAbstractKeyFactory.java
 PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java 
5008f15 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKeyFactory.java
 PRE-CREATION 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseDefaultKeyFactory.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseKeyFactory.java 
PRE-CREATION 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseScanRange.java 
PRE-CREATION 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 
b64590d 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java 
4fe1b1b 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
 142bfd8 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java fc40195 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java 
13c344b 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java 
PRE-CREATION 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory2.java 
PRE-CREATION 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
7c4fc9f 
  hbase-handler/src/test/queries/positive/hbase_custom_key.q PRE-CREATION 
  hbase-handler/src/test/queries/positive/hbase_custom_key2.q PRE-CREATION 
  hbase-handler/src/test/results/positive/hbase_custom_key.q.out PRE-CREATION 
  hbase-handler/src/test/results/positive/hbase_custom_key2.q.out PRE-CREATION 
  itests/util/pom.xml e9720df 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e52d364 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 
d39ee2e 
  ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java 5f1329c 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java f0c0ecf 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStoragePredicateHandler.java 
9f35575 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java e50026b 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java ecb82d7 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java c0a8269 
  serde/src/java/org/apache/hadoop/hive/serde2/StructObject.java PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/StructObjectBaseInspector.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStructBase.java 
1fd6853 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java 10f4c05 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java 3334dff 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
82c1263 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java 8a1ea46 
  
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
 8a5386a 
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryObject.java 
598683f 
  serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java 
caf3517 

Diff: https://reviews.apache.org/r/18179/diff/


Testing
---


Thanks,

Navis Ryu

[jira] [Updated] (HIVE-6411) Support more generic way of using composite key for HBaseHandler

2014-03-31 Thread Navis (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6411:


Attachment: HIVE-6411.8.patch.txt

 Support more generic way of using composite key for HBaseHandler
 

 Key: HIVE-6411
 URL: https://issues.apache.org/jira/browse/HIVE-6411
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-6411.1.patch.txt, HIVE-6411.2.patch.txt, 
 HIVE-6411.3.patch.txt, HIVE-6411.4.patch.txt, HIVE-6411.5.patch.txt, 
 HIVE-6411.6.patch.txt, HIVE-6411.7.patch.txt, HIVE-6411.8.patch.txt


 HIVE-2599 introduced using custom object for the row key. But it forces key 
 objects to extend HBaseCompositeKey, which is again extension of LazyStruct. 
 If user provides proper Object and OI, we can replace internal key and keyOI 
 with those. 
 Initial implementation is based on factory interface.
 {code}
 public interface HBaseKeyFactory {
   void init(SerDeParameters parameters, Properties properties) throws 
 SerDeException;
   ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
   LazyObjectBase createObject(ObjectInspector inspector) throws 
 SerDeException;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6797) Add protection against divide by zero in stats annotation


[ 
https://issues.apache.org/jira/browse/HIVE-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955964#comment-13955964
 ] 

Harish Butani commented on HIVE-6797:
-

+1

 Add protection against divide by zero in stats annotation
 -

 Key: HIVE-6797
 URL: https://issues.apache.org/jira/browse/HIVE-6797
 Project: Hive
  Issue Type: Sub-task
  Components: Query Processor, Statistics
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J
 Fix For: 0.13.0

 Attachments: HIVE-6797.1.patch


 In stats annotation, the denominator computation in join operator is not 
 protected for divide by zero exception. It will be an issue when NDV (count 
 distinct) updated by updateStats() becomes 0. This patch adds protection in 
 updateStats() method to prevent divide-by-zero in downstream operators.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6802) Fix metastore.thrift: add partition_columns.types constant


 [ 
https://issues.apache.org/jira/browse/HIVE-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6802:


Status: Patch Available  (was: Open)

 Fix metastore.thrift: add partition_columns.types constant
 --

 Key: HIVE-6802
 URL: https://issues.apache.org/jira/browse/HIVE-6802
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
 Attachments: HIVE-6802.1.patch


 HIVE-6642 edited the hive_metastoreConstants.java genned file. 
 Need to add constant to thrift file and regen thrift classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6786) Off by one error in ORC PPD


[ 
https://issues.apache.org/jira/browse/HIVE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955967#comment-13955967
 ] 

Harish Butani commented on HIVE-6786:
-

+1 for 0.13

 Off by one error in ORC PPD 
 

 Key: HIVE-6786
 URL: https://issues.apache.org/jira/browse/HIVE-6786
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Gopal V
Assignee: Prasanth J
Priority: Critical
 Fix For: 0.13.0

 Attachments: HIVE-6786.1.patch


 Turning on ORC PPD makes split computation fail for a 10Tb benchmark.
 Narrowed down to the following code fragment
 https://github.com/apache/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java#L757
 {code}
 includeStripe[i] = (i  stripeStats.size()) ||
 isStripeSatisfyPredicate(stripeStats.get(i), sarg,
  filterColumns);
 {code}
 I would guess that should be a =, but [~prasanth_j], can you comment if that 
 is the right fix?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6430) MapJoin hash table has large memory overhead

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955969#comment-13955969
 ] 

Sergey Shelukhin commented on HIVE-6430:


[~gopalv] do you want to finish review when you have time?

 MapJoin hash table has large memory overhead
 

 Key: HIVE-6430
 URL: https://issues.apache.org/jira/browse/HIVE-6430
 Project: Hive
  Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HIVE-6430.01.patch, HIVE-6430.02.patch, 
 HIVE-6430.03.patch, HIVE-6430.04.patch, HIVE-6430.05.patch, 
 HIVE-6430.06.patch, HIVE-6430.patch


 Right now, in some queries, I see that storing e.g. 4 ints (2 for key and 2 
 for row) can take several hundred bytes, which is ridiculous. I am reducing 
 the size of MJKey and MJRowContainer in other jiras, but in general we don't 
 need to have java hash table there.  We can either use primitive-friendly 
 hashtable like the one from HPPC (Apache-licenced), or some variation, to map 
 primitive keys to single row storage structure without an object per row 
 (similar to vectorization).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6802) Fix metastore.thrift: add partition_columns.types constant


[ 
https://issues.apache.org/jira/browse/HIVE-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955970#comment-13955970
 ] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-6802:
-

+1. Thanks [~rhbutani] for incorporating this missing change.

 Fix metastore.thrift: add partition_columns.types constant
 --

 Key: HIVE-6802
 URL: https://issues.apache.org/jira/browse/HIVE-6802
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
 Attachments: HIVE-6802.1.patch


 HIVE-6642 edited the hive_metastoreConstants.java genned file. 
 Need to add constant to thrift file and regen thrift classes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6749) Turn hive.auto.convert.join.use.nonstaged off by default

2014-03-31 Thread Navis (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955973#comment-13955973
 ] 

Navis commented on HIVE-6749:
-

Distributing small aliases(about 500M max) through DistributedCache makes busy 
works between DNs, and it was not good for the whole system. But +1 for the 
idea. It's not matured, yet.

 Turn hive.auto.convert.join.use.nonstaged off by default
 

 Key: HIVE-6749
 URL: https://issues.apache.org/jira/browse/HIVE-6749
 Project: Hive
  Issue Type: Bug
  Components: Configuration
Affects Versions: 0.13.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan
 Attachments: HIVE-6749.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HIVE-6769) HCatStorer can initialize SerDe with less columns than table definition if PigSchema has less columns

2014-03-31 Thread Eugene Koifman (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-6769.
--

Resolution: Won't Fix

no change required.  On read the SerDe should provide the 'missing' column with 
value NULL.  This would be the same case as when a table is created with 5 
columns and data written to it.  Then 2 columns are added.  Then a 'select * 
form T' should still work.

 HCatStorer can initialize SerDe with less columns than table definition if 
 PigSchema has less columns
 -

 Key: HIVE-6769
 URL: https://issues.apache.org/jira/browse/HIVE-6769
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Reporter: Jason Dere

 You should be able to see this in TestHCatStorer.testStoreFuncAllSimpleTypes. 
  First a Hive table is created with 8 columns.  Then HCatStorer is used to 
 populate the table, but Pig relation only has data for the first 7 columns of 
 the table. When HCat initializes the SerDe for writing the table, the schema 
 only contains 7 columns rather than the 8 that the table should have. This 
 test works without any errors, but gets an error if the table is changed to 
 ORC storage (which can be fixed by HIVE-4975).
 HCat should initialize the Serde with the full number of columns in the 
 table, and should fill in NULLs for any columns which are not provided by Pig 
 relation.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6778) ql/src/test/queries/clientpositive/pcr.q covers the test which generate 1.0 =1 predicate in partition pruner.

2014-03-31 Thread Hari Sankar Sivarama Subramaniyan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6778:


Attachment: HIVE-6778.1.patch

 ql/src/test/queries/clientpositive/pcr.q covers the test which generate 1.0 
 =1 predicate in partition pruner. 
 --

 Key: HIVE-6778
 URL: https://issues.apache.org/jira/browse/HIVE-6778
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Harish Butani
 Attachments: HIVE-6778.1.patch


 select key, value, ds from pcr_foo where (ds % 2 == 1);
 ql/src/test/queries/clientpositive/pcr.q
 The test generates 1.0==1 predicate in the pruner which cannot be evaluated 
 since a double cannot be converted to int.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6329) Support column level encryption/decryption

2014-03-31 Thread Navis (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6329:


Attachment: HIVE-6329.8.patch.txt

 Support column level encryption/decryption
 --

 Key: HIVE-6329
 URL: https://issues.apache.org/jira/browse/HIVE-6329
 Project: Hive
  Issue Type: New Feature
  Components: Security, Serializers/Deserializers
Reporter: Navis
Assignee: Navis
Priority: Minor
 Attachments: HIVE-6329.1.patch.txt, HIVE-6329.2.patch.txt, 
 HIVE-6329.3.patch.txt, HIVE-6329.4.patch.txt, HIVE-6329.5.patch.txt, 
 HIVE-6329.6.patch.txt, HIVE-6329.7.patch.txt, HIVE-6329.8.patch.txt


 Receiving some requirements on encryption recently but hive is not supporting 
 it. Before the full implementation via HIVE-5207, this might be useful for 
 some cases.
 {noformat}
 hive create table encode_test(id int, name STRING, phone STRING, address 
 STRING) 
  ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
  WITH SERDEPROPERTIES ('column.encode.indices'='2,3', 
 'column.encode.classname'='org.apache.hadoop.hive.serde2.Base64WriteOnly') 
 STORED AS TEXTFILE;
 OK
 Time taken: 0.584 seconds
 hive insert into table encode_test select 
 100,'navis','010--','Seoul, Seocho' from src tablesample (1 rows);
 ..
 OK
 Time taken: 5.121 seconds
 hive select * from encode_test;
 OK
 100   navis MDEwLTAwMDAtMDAwMA==  U2VvdWwsIFNlb2Nobw==
 Time taken: 0.078 seconds, Fetched: 1 row(s)
 hive 
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 17652: Support column level encryption/decryption

2014-03-31 Thread Navis Ryu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17652/
---

(Updated April 1, 2014, 2:01 a.m.)


Review request for hive.


Changes
---

Rebased to trunk


Bugs: HIVE-6329
https://issues.apache.org/jira/browse/HIVE-6329


Repository: hive-git


Description
---

Receiving some requirements on encryption recently but hive is not supporting 
it. Before the full implementation via HIVE-5207, this might be useful for some 
cases.

{noformat}
hive create table encode_test(id int, name STRING, phone STRING, address 
STRING) 
 ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
 WITH SERDEPROPERTIES ('column.encode.indices'='2,3', 
'column.encode.classname'='org.apache.hadoop.hive.serde2.Base64WriteOnly') 
STORED AS TEXTFILE;
OK
Time taken: 0.584 seconds
hive insert into table encode_test select 100,'navis','010--','Seoul, 
Seocho' from src tablesample (1 rows);
..
OK
Time taken: 5.121 seconds
hive select * from encode_test;
OK
100 navis MDEwLTAwMDAtMDAwMA==  U2VvdWwsIFNlb2Nobw==
Time taken: 0.078 seconds, Fetched: 1 row(s)
hive 
{noformat}


Diffs (updated)
-

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseCompositeKey.java 
5008f15 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java 
b64590d 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java 
cedef10 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java fc40195 
  
hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestCompositeKey.java 
13c344b 
  hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java 
7c4fc9f 
  hbase-handler/src/test/queries/positive/hbase_column_encoding.q PRE-CREATION 
  hbase-handler/src/test/results/positive/hbase_column_encoding.q.out 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java c4268c1 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java 293b74e 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ArrayWritableObjectInspector.java
 bb02bab 
  ql/src/test/queries/clientpositive/column_encoding.q PRE-CREATION 
  ql/src/test/results/clientpositive/column_encoding.q.out PRE-CREATION 
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
 515cf25 
  serde/src/java/org/apache/hadoop/hive/serde2/AbstractFieldRewriter.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/Base64Rewriter.java PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/Base64WriteOnly.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29 
  serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 179f9b5 
  serde/src/java/org/apache/hadoop/hive/serde2/FieldRewritable.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/FieldRewriter.java PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/NullStructSerDe.java dba5e33 
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStruct.java 
157600e 
  serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarStructBase.java 
1fd6853 
  
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarStruct.java
 6d9715a 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyArray.java ce0cfb3 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBinary.java ae12f20 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyBoolean.java 42cb43c 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyByte.java a3b8f76 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyDate.java 7af2374 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyDouble.java 05ca4e9 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFloat.java 37676d1 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveChar.java ef469eb 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveDecimal.java 
78cc381 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyHiveVarchar.java 
bc8d41e 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyInteger.java ad82ebf 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyLong.java a9779a0 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyMap.java 686fc76 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyNonPrimitive.java 
e4cffc9 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObject.java 10f4c05 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyObjectBase.java 3334dff 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyPrimitive.java 7d23c46 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyShort.java f04e131 
  serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
82c1263

[jira] [Commented] (HIVE-6778) ql/src/test/queries/clientpositive/pcr.q covers the test which generate 1.0 =1 predicate in partition pruner.


[ 
https://issues.apache.org/jira/browse/HIVE-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13956022#comment-13956022
 ] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-6778:
-

+1

 ql/src/test/queries/clientpositive/pcr.q covers the test which generate 1.0 
 =1 predicate in partition pruner. 
 --

 Key: HIVE-6778
 URL: https://issues.apache.org/jira/browse/HIVE-6778
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Harish Butani
 Attachments: HIVE-6778.1.patch


 select key, value, ds from pcr_foo where (ds % 2 == 1);
 ql/src/test/queries/clientpositive/pcr.q
 The test generates 1.0==1 predicate in the pruner which cannot be evaluated 
 since a double cannot be converted to int.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6766) HCatLoader always returns Char datatype with maxlength(255) when table format is ORC


 [ 
https://issues.apache.org/jira/browse/HIVE-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6766:


Attachment: HIVE-6766.1.patch

HIVE-6766.1.patch - attaching file again for precommit tests 

 HCatLoader always returns Char datatype with maxlength(255)  when table 
 format is ORC
 -

 Key: HIVE-6766
 URL: https://issues.apache.org/jira/browse/HIVE-6766
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.13.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Priority: Critical
 Attachments: HIVE-6766.1.patch, HIVE-6766.patch


 attached patch contains
 org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer#testWriteChar()
 which shows that char(5) value written to Hive (ORC) table using HCatStorer 
 will come back as char(255) when read with HCatLoader.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6766) HCatLoader always returns Char datatype with maxlength(255) when table format is ORC