[jira] [Commented] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932916#comment-13932916
 ] 

Gopal V commented on HIVE-6455:
---

{code}
create table session ( ... )
partitioned by (start_dt date);

insert overwrite table session partition(start_dt)  from session_raw;
{code}

with the patch .15

> Scalable dynamic partitioning and bucketing optimization
> 
>
> Key: HIVE-6455
> URL: https://issues.apache.org/jira/browse/HIVE-6455
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, 
> HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.11.patch, 
> HIVE-6455.12.patch, HIVE-6455.13.patch, HIVE-6455.13.patch, 
> HIVE-6455.14.patch, HIVE-6455.15.patch, HIVE-6455.2.patch, HIVE-6455.3.patch, 
> HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, 
> HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
>
>
> The current implementation of dynamic partition works by keeping at least one 
> record writer open per dynamic partition directory. In case of bucketing 
> there can be multispray file writers which further adds up to the number of 
> open record writers. The record writers of column oriented file format (like 
> ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or 
> compression buffers) open all the time to buffer up the rows and compress 
> them before flushing it to disk. Since these buffers are maintained per 
> column basis the amount of constant memory that will required at runtime 
> increases as the number of partitions and number of columns per partition 
> increases. This often leads to OutOfMemory (OOM) exception in mappers or 
> reducers depending on the number of open record writers. Users often tune the 
> JVM heapsize (runtime memory) to get over such OOM issues. 
> With this optimization, the dynamic partition columns and bucketing columns 
> (in case of bucketed tables) are sorted before being fed to the reducers. 
> Since the partitioning and bucketing columns are sorted, each reducers can 
> keep only one record writer open at any time thereby reducing the memory 
> pressure on the reducers. This optimization is highly scalable as the number 
> of partition and number of columns per partition increases at the cost of 
> sorting the columns.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932915#comment-13932915
 ] 

Gopal V commented on HIVE-6455:
---

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.partialCopyToStandardObject(ObjectInspectorUtils.java:213)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:816)
at 
org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:497)
at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:520)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.processKeyValues(ReduceRecordProcessor.java:276)
... 8 more
{code}

for a partitioned insert in Tez.

> Scalable dynamic partitioning and bucketing optimization
> 
>
> Key: HIVE-6455
> URL: https://issues.apache.org/jira/browse/HIVE-6455
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, 
> HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.11.patch, 
> HIVE-6455.12.patch, HIVE-6455.13.patch, HIVE-6455.13.patch, 
> HIVE-6455.14.patch, HIVE-6455.15.patch, HIVE-6455.2.patch, HIVE-6455.3.patch, 
> HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, 
> HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
>
>
> The current implementation of dynamic partition works by keeping at least one 
> record writer open per dynamic partition directory. In case of bucketing 
> there can be multispray file writers which further adds up to the number of 
> open record writers. The record writers of column oriented file format (like 
> ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or 
> compression buffers) open all the time to buffer up the rows and compress 
> them before flushing it to disk. Since these buffers are maintained per 
> column basis the amount of constant memory that will required at runtime 
> increases as the number of partitions and number of columns per partition 
> increases. This often leads to OutOfMemory (OOM) exception in mappers or 
> reducers depending on the number of open record writers. Users often tune the 
> JVM heapsize (runtime memory) to get over such OOM issues. 
> With this optimization, the dynamic partition columns and bucketing columns 
> (in case of bucketed tables) are sorted before being fed to the reducers. 
> Since the partitioning and bucketing columns are sorted, each reducers can 
> keep only one record writer open at any time thereby reducing the memory 
> pressure on the reducers. This optimization is highly scalable as the number 
> of partition and number of columns per partition increases at the cost of 
> sorting the columns.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory overhead

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review36939
---



ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java


The ref longs are not just indexes, they contain bits from various 
components.



ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java






ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java


reserve -ve bytes?



ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java


Really? :)



ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java


Rehashing involves calculating hashCodes of all keys - this is very 
expensive CPU wise.


- Gopal V


On March 12, 2014, 2:30 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> ---
> 
> (Updated March 12, 2014, 2:30 a.m.)
> 
> 
> Review request for hive, Gopal V and Gunther Hagleitner.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> See JIRA
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38 
>   ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> 170e8c0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java
>  8854b19 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 
> 61545b5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 
> a00aab3 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java
>  008a8db 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java
>  a8cb1ae 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
>  55b7415 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
> 6ecbcf7 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java
>  65e3779 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java
>  755d783 
>   ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6 
>   ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984 
>   ql/src/test/results/clientpositive/tez/ctas.q.out aaac91c 
>   ql/src/test/results/clientpositive/tez/groupby3.q.out cb8f0e7 
>   ql/src/test/results/clientpositive/tez/insert1.q.out 7b8819d 
>   ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 7d6b1c7 
>   ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out d420caf 
>   ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650 
>   ql/src/test/results/clientpositive/tez/subquery_in.q.out c66602b 
>   serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29 
>   serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java
>  bab505e 
>   serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd 
>   serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java 
> a99c7b4 
>   serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 
> 435d6c6 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 
> b188c3f 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 
> 6c14081 
> 
> Diff: https://reviews.apache.org/r/18936/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

[jira] [Commented] (HIVE-6646) Error in txn handler SQL

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932897#comment-13932897
 ] 

Ashutosh Chauhan commented on HIVE-6646:


+1

> Error in txn handler SQL
> 
>
> Key: HIVE-6646
> URL: https://issues.apache.org/jira/browse/HIVE-6646
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6646.patch
>
>
> There are a few places where the SQL in TxnHandler.java works in Derby but 
> not in MySQL due to differences in the way they parse tokens.  Adding spaces 
> to all such places addresses the issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6648) Permissions are not inherited correctly when tables have multiple partition columns

[Henry Robinson (JIRA)]

Henry Robinson created HIVE-6648:


 Summary: Permissions are not inherited correctly when tables have 
multiple partition columns
 Key: HIVE-6648
 URL: https://issues.apache.org/jira/browse/HIVE-6648
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Henry Robinson


{{Warehouse.mkdirs()}} always looks at the immediate parent of the path that it 
creates when determining what permissions to inherit. However, it may have 
created that parent directory as well, in which case it will have the default 
permissions and will not have inherited them.

This is a problem when performing an {{INSERT}} into a table with more than one 
partition column. E.g., in an empty table:

{{INSERT INTO TABLE tbl PARTITION(p1=1, p2=2) ... }}

A new subdirectory /p1=1/p2=2  will be created, and with permission inheritance 
(per HIVE-2504) enabled, the intention is presumably for both new directories 
to inherit the root table dir's permissions. However, {{mkdirs()}} will only 
set the permission of the leaf directory (i.e. /p2=2/), and then only to the 
permissions of /p1=1/, which was just created.

{code}
public boolean mkdirs(Path f) throws MetaException {
FileSystem fs = null;
try {
  fs = getFs(f);
  LOG.debug("Creating directory if it doesn't exist: " + f);
  //Check if the directory already exists. We want to change the permission
  //to that of the parent directory only for newly created directories.
  if (this.inheritPerms) {
try {
  return fs.getFileStatus(f).isDir();
} catch (FileNotFoundException ignore) {
}
  }
  boolean success = fs.mkdirs(f);
  if (this.inheritPerms && success) {
// Set the permission of parent directory.
// HNR: This is the bug - getParent() may refer to a just-created 
directory.
fs.setPermission(f, fs.getFileStatus(f.getParent()).getPermission());
  }
  return success;
} catch (IOException e) {
  closeFs(fs);
  MetaStoreUtils.logAndThrowMetaException(e);
}
return false;
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6647) Bump the thrift api version to V7 for HiveServer2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932884#comment-13932884
 ] 

Thejas M Nair commented on HIVE-6647:
-

Hi [~prasadm], Will you able to contribute a patch for this change ?


> Bump the thrift api version to V7 for HiveServer2
> -
>
> Key: HIVE-6647
> URL: https://issues.apache.org/jira/browse/HIVE-6647
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Priority: Blocker
> Fix For: 0.13.0
>
>
> HIVE-5155 added new api for delegation token support. Per the convention 
> followed till now, we should update the version to 7. 
> Marking it as blocker for 13. cc [~prasadm] [~thejas]



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6647) Bump the thrift api version to V7 for HiveServer2

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-6647:
--

 Summary: Bump the thrift api version to V7 for HiveServer2
 Key: HIVE-6647
 URL: https://issues.apache.org/jira/browse/HIVE-6647
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Priority: Blocker
 Fix For: 0.13.0


HIVE-5155 added new api for delegation token support. Per the convention 
followed till now, we should update the version to 7. 

Marking it as blocker for 13. cc [~prasadm] [~thejas]



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6629) Change SET ROLE NONE to SET ROLE ALL

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932878#comment-13932878
 ] 

Thejas M Nair commented on HIVE-6629:
-

Brock, I have added some comments on review board.


> Change SET ROLE NONE to SET ROLE ALL
> 
>
> Key: HIVE-6629
> URL: https://issues.apache.org/jira/browse/HIVE-6629
> Project: Hive
>  Issue Type: Task
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: HIVE-6629.patch
>
>
> I understand this is well after it's been implemented. However, I think we 
> need to discuss the counter-intuitive SET ROLE semantics before we release 
> 0.13.
> The new Hive Authz work implements "SET ROLE NONE" similar to MySQL meaning 
> that "SET ROLE NONE" actually sets your role to the default. This is 
> extremely counter-intuitive.
> * 
> [Oracle|http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_10004.htm]
>  
> * 
> [Informix|http://pic.dhe.ibm.com/infocenter/idshelp/v117/index.jsp?topic=%2Fcom.ibm.sqls.doc%2Fids_sqs_1186.htm]
> * [Vertica|https://my.vertica.com/docs/5.1.6/HTML/index.htm#15645.htm]
> * 
> [SAP|http://www.sapdb.org/htmhelp/44/a17998442911d3a98200a0c9449261/content.htm]
> All of the widely deployed databases above have "SET ROLE NONE" disable all 
> privileges. Those databases have the intuitive "SET ROLE ALL" or "SET ROLE 
> DEFAULT" enable all privileges.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HIVE-6629) Change SET ROLE NONE to SET ROLE ALL

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932878#comment-13932878
 ] 

Thejas M Nair edited comment on HIVE-6629 at 3/13/14 5:19 AM:
--

Brock, Thanks for the patch!
I have added some comments on review board.



was (Author: thejas):
Brock, I have added some comments on review board.


> Change SET ROLE NONE to SET ROLE ALL
> 
>
> Key: HIVE-6629
> URL: https://issues.apache.org/jira/browse/HIVE-6629
> Project: Hive
>  Issue Type: Task
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: HIVE-6629.patch
>
>
> I understand this is well after it's been implemented. However, I think we 
> need to discuss the counter-intuitive SET ROLE semantics before we release 
> 0.13.
> The new Hive Authz work implements "SET ROLE NONE" similar to MySQL meaning 
> that "SET ROLE NONE" actually sets your role to the default. This is 
> extremely counter-intuitive.
> * 
> [Oracle|http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_10004.htm]
>  
> * 
> [Informix|http://pic.dhe.ibm.com/infocenter/idshelp/v117/index.jsp?topic=%2Fcom.ibm.sqls.doc%2Fids_sqs_1186.htm]
> * [Vertica|https://my.vertica.com/docs/5.1.6/HTML/index.htm#15645.htm]
> * 
> [SAP|http://www.sapdb.org/htmhelp/44/a17998442911d3a98200a0c9449261/content.htm]
> All of the widely deployed databases above have "SET ROLE NONE" disable all 
> privileges. Those databases have the intuitive "SET ROLE ALL" or "SET ROLE 
> DEFAULT" enable all privileges.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 19150: HIVE-6629 - change SET ROLE NONE to SET ROLE ALL

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19150/#review37028
---



ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g


These should be added to nonReserved rule in IdentifiersParser.g . That 
enables them to be used as identifiers as well.

Otherwise, it is a backward incompatible change. These keywords can no 
longer be used as columns or table names. It can break existing queries.



ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g


Once these are allowed as identifiers, they can be used as role names in 
create role.
To prevent that that we need to have an explicit check in 
SQLStdHiveAccessController.createRole().
Can you also add negative tests for create role with these role names ?




ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g


Once it is possible to use the ALL keyword as identifier, this rule is 
redundant. I expect it to result in an additional antlr warning.
We should remove it. 



- Thejas Nair


On March 12, 2014, 7:23 p.m., Brock Noland wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19150/
> ---
> 
> (Updated March 12, 2014, 7:23 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1) Changes SET ROLE NONE to SET ROLE ALL.
> 2) Reserves the keywords NONE and DEFAULT.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g cdfa300 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g a74da0e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLStdHiveAccessController.java
>  f69e41b 
>   ql/src/test/queries/clientnegative/authorization_disallow_transform.q 
> bae8dc0 
>   ql/src/test/queries/clientnegative/authorization_drop_role_no_admin.q 
> 80346d4 
>   ql/src/test/queries/clientpositive/authorization_set_show_current_role.q 
> 7fe8e29 
>   ql/src/test/results/clientnegative/authorization_disallow_transform.q.out 
> 044b19e 
>   ql/src/test/results/clientnegative/authorization_drop_role_no_admin.q.out 
> d2cc1dd 
>   
> ql/src/test/results/clientpositive/authorization_set_show_current_role.q.out 
> 5c39a06 
> 
> Diff: https://reviews.apache.org/r/19150/diff/
> 
> 
> Testing
> ---
> 
> All tests which call SET ROLE were manually tested.
> 
> 
> Thanks,
> 
> Brock Noland
> 
>

[jira] [Commented] (HIVE-6222) Make Vector Group By operator abandon grouping if too many distinct keys

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932874#comment-13932874
 ] 

Jitendra Nath Pandey commented on HIVE-6222:


+1

> Make Vector Group By operator abandon grouping if too many distinct keys
> 
>
> Key: HIVE-6222
> URL: https://issues.apache.org/jira/browse/HIVE-6222
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
>  Labels: vectorization
> Attachments: HIVE-6222.1.patch, HIVE-6222.2.patch, HIVE-6222.3.patch
>
>
> Row mode GBY is becoming a pass-through if not enough aggregation occurs on 
> the map side, relying on the shuffle+reduce side to do the work. Have VGBY do 
> the same.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6613) Control when spcific Inputs / Outputs are started

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932870#comment-13932870
 ] 

Siddharth Seth commented on HIVE-6613:
--

Thanks for taking a look.

bq. Can you avoid creating a conf in TezCacheAccess? Maybe just pass it in 
get().
Was doing this once in the static block to avoid having to use a Configuration 
instance to access this class. TezCacheAccess is only supposed to be used with 
Tez. I could skip the factory all together and instantiate the Tez cache 
directly ? (The Configuration creation in this case should be very cheap since 
it isn't accessing external files)

bq. Have you considered adding the input to the cache key instead of using a 
Set? 
The set just groups the fact that they're cached together. I can use individual 
keys if you think that's better. That will get rid of the lock - since the 
primary purpose is to control the set creation.

bq. You can drop the getLocalWork check in the tez hashtable loader. Tez 
doesn't have local work.
bq. The javadoc of the init function needs to be updated with your changes.
Will fix

> Control when spcific Inputs / Outputs are started
> -
>
> Key: HIVE-6613
> URL: https://issues.apache.org/jira/browse/HIVE-6613
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-6613.1.txt
>
>
> When running with Tez - a couple of enhancement are possible
> 1) Avoid re-fetching data in case of MapJoins - since the data is likely to 
> be cached after the first run (container re-use for the same query)
> 2) Start Outputs only after required Inputs are ready - specifically useful 
> in case of Reduce - where shuffle requires a large memory, and the Output (if 
> it's a sorted output) also requires a fair amount of memory.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6060) Define API for RecordUpdater and UpdateReader

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-6060:


Attachment: HIVE-6060.patch

> Define API for RecordUpdater and UpdateReader
> -
>
> Key: HIVE-6060
> URL: https://issues.apache.org/jira/browse/HIVE-6060
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-6060.patch, HIVE-6060.patch, acid-io.patch, 
> h-5317.patch, h-5317.patch, h-5317.patch, h-6060.patch, h-6060.patch
>
>
> We need to define some new APIs for how Hive interacts with the file formats 
> since it needs to be much richer than the current RecordReader and 
> RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6060) Define API for RecordUpdater and UpdateReader

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-6060:


Attachment: (was: HIVE-6060.patch)

> Define API for RecordUpdater and UpdateReader
> -
>
> Key: HIVE-6060
> URL: https://issues.apache.org/jira/browse/HIVE-6060
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-6060.patch, HIVE-6060.patch, acid-io.patch, 
> h-5317.patch, h-5317.patch, h-5317.patch, h-6060.patch, h-6060.patch
>
>
> We need to define some new APIs for how Hive interacts with the file formats 
> since it needs to be much richer than the current RecordReader and 
> RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6060) Define API for RecordUpdater and UpdateReader

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-6060:


Attachment: HIVE-6060.patch

Addressed Sergey's comments.

> Define API for RecordUpdater and UpdateReader
> -
>
> Key: HIVE-6060
> URL: https://issues.apache.org/jira/browse/HIVE-6060
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-6060.patch, HIVE-6060.patch, acid-io.patch, 
> h-5317.patch, h-5317.patch, h-5317.patch, h-6060.patch, h-6060.patch
>
>
> We need to define some new APIs for how Hive interacts with the file formats 
> since it needs to be much richer than the current RecordReader and 
> RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6635:
-

Status: Open  (was: Patch Available)

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932839#comment-13932839
 ] 

Alan Gates commented on HIVE-6635:
--

Have we solved all the places that DDL operations can take an hour plus?  And 
is this same job helper used by Tez or do we need to do it separately in that 
case?

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6646) Error in txn handler SQL

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6646:
-

Attachment: HIVE-6646.patch

> Error in txn handler SQL
> 
>
> Key: HIVE-6646
> URL: https://issues.apache.org/jira/browse/HIVE-6646
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6646.patch
>
>
> There are a few places where the SQL in TxnHandler.java works in Derby but 
> not in MySQL due to differences in the way they parse tokens.  Adding spaces 
> to all such places addresses the issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6646) Error in txn handler SQL

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6646:
-

Status: Patch Available  (was: Open)

> Error in txn handler SQL
> 
>
> Key: HIVE-6646
> URL: https://issues.apache.org/jira/browse/HIVE-6646
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6646.patch
>
>
> There are a few places where the SQL in TxnHandler.java works in Derby but 
> not in MySQL due to differences in the way they parse tokens.  Adding spaces 
> to all such places addresses the issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-1662) Add file pruning into Hive.

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932793#comment-13932793
 ] 

Navis commented on HIVE-1662:
-

I don't think this ever be committed.

> Add file pruning into Hive.
> ---
>
> Key: HIVE-1662
> URL: https://issues.apache.org/jira/browse/HIVE-1662
> Project: Hive
>  Issue Type: New Feature
>Reporter: He Yongqiang
>Assignee: Navis
> Attachments: HIVE-1662.10.patch.txt, HIVE-1662.11.patch.txt, 
> HIVE-1662.12.patch.txt, HIVE-1662.8.patch.txt, HIVE-1662.9.patch.txt, 
> HIVE-1662.D8391.1.patch, HIVE-1662.D8391.2.patch, HIVE-1662.D8391.3.patch, 
> HIVE-1662.D8391.4.patch, HIVE-1662.D8391.5.patch, HIVE-1662.D8391.6.patch, 
> HIVE-1662.D8391.7.patch
>
>
> now hive support filename virtual column. 
> if a file name filter presents in a query, hive should be able to only add 
> files which passed the filter to input paths.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6037) Synchronize HiveConf with hive-default.xml.template and support show conf

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932789#comment-13932789
 ] 

Lefty Leverenz commented on HIVE-6037:
--

[~navis], the new patch doesn't seem to have any line breaks.

> Synchronize HiveConf with hive-default.xml.template and support show conf
> -
>
> Key: HIVE-6037
> URL: https://issues.apache.org/jira/browse/HIVE-6037
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: CHIVE-6037.3.patch.txt, HIVE-6037-0.13.0, 
> HIVE-6037.1.patch.txt, HIVE-6037.10.patch.txt, HIVE-6037.11.patch.txt, 
> HIVE-6037.12.patch.txt, HIVE-6037.14.patch.txt, HIVE-6037.15.patch.txt, 
> HIVE-6037.16.patch.txt, HIVE-6037.17.patch, HIVE-6037.2.patch.txt, 
> HIVE-6037.4.patch.txt, HIVE-6037.5.patch.txt, HIVE-6037.6.patch.txt, 
> HIVE-6037.7.patch.txt, HIVE-6037.8.patch.txt, HIVE-6037.9.patch.txt, 
> HIVE-6037.patch
>
>
> see HIVE-5879



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6037) Synchronize HiveConf with hive-default.xml.template and support show conf

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6037:


Attachment: HIVE-6037-0.13.0

Patch for 0.13.0

> Synchronize HiveConf with hive-default.xml.template and support show conf
> -
>
> Key: HIVE-6037
> URL: https://issues.apache.org/jira/browse/HIVE-6037
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: CHIVE-6037.3.patch.txt, HIVE-6037-0.13.0, 
> HIVE-6037.1.patch.txt, HIVE-6037.10.patch.txt, HIVE-6037.11.patch.txt, 
> HIVE-6037.12.patch.txt, HIVE-6037.14.patch.txt, HIVE-6037.15.patch.txt, 
> HIVE-6037.16.patch.txt, HIVE-6037.17.patch, HIVE-6037.2.patch.txt, 
> HIVE-6037.4.patch.txt, HIVE-6037.5.patch.txt, HIVE-6037.6.patch.txt, 
> HIVE-6037.7.patch.txt, HIVE-6037.8.patch.txt, HIVE-6037.9.patch.txt, 
> HIVE-6037.patch
>
>
> see HIVE-5879



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6644) document TestStreaming_2 e2e test case for webhcat

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-6644:
-

Status: Patch Available  (was: Open)

> document TestStreaming_2 e2e test case for webhcat
> --
>
> Key: HIVE-6644
> URL: https://issues.apache.org/jira/browse/HIVE-6644
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-6644.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6644) document TestStreaming_2 e2e test case for webhcat

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-6644:
-

Attachment: HIVE-6644.patch

no pre commit test

> document TestStreaming_2 e2e test case for webhcat
> --
>
> Key: HIVE-6644
> URL: https://issues.apache.org/jira/browse/HIVE-6644
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-6644.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6646) Error in txn handler SQL

[Alan Gates (JIRA)]

Alan Gates created HIVE-6646:


 Summary: Error in txn handler SQL
 Key: HIVE-6646
 URL: https://issues.apache.org/jira/browse/HIVE-6646
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 0.13.0


There are a few places where the SQL in TxnHandler.java works in Derby but not 
in MySQL due to differences in the way they parse tokens.  Adding spaces to all 
such places addresses the issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6645) to_date()/to_unix_timestamp() fail with NPE if input is null

[Jason Dere (JIRA)]

Jason Dere created HIVE-6645:


 Summary: to_date()/to_unix_timestamp() fail with NPE if input is 
null
 Key: HIVE-6645
 URL: https://issues.apache.org/jira/browse/HIVE-6645
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Jason Dere
Assignee: Jason Dere


{noformat}
hive> describe tab2;
Query ID = jdere_20140312185454_e3ed213e-8b3a-4963-b815-19965edad587
OK
c1  timestamp   None
Time taken: 0.155 seconds, Fetched: 1 row(s)
hive> select * from tab2;
Query ID = jdere_20140312185454_8a009070-df79-45de-8642-e85668a378d7
OK
NULL
NULL
NULL
NULL
NULL
Time taken: 0.067 seconds, Fetched: 5 row(s)

hive> select to_unix_timestamp(c1) from tab2;   
hive> select to_date(c1) from tab2;  
{noformat}

Fails with errors like:

{noformat}
java.lang.Exception: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row {"c1":null}
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:401)
Caused by: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row {"c1":null}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:233)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row {"c1":null}
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
... 10 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating 
to_date(c1)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
... 11 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFDate.evaluate(GenericUDFDate.java:106)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
... 15 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-1662) Add file pruning into Hive.

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932776#comment-13932776
 ] 

Lefty Leverenz commented on HIVE-1662:
--

*hive.optimize.ppd.vc.filename* still needs documentation (release note and 
hive-default.xml.template, unless this commits after HIVE-6037 in which case 
HiveConf.java will be changed and the doc goes in the parameter definition).

> Add file pruning into Hive.
> ---
>
> Key: HIVE-1662
> URL: https://issues.apache.org/jira/browse/HIVE-1662
> Project: Hive
>  Issue Type: New Feature
>Reporter: He Yongqiang
>Assignee: Navis
> Attachments: HIVE-1662.10.patch.txt, HIVE-1662.11.patch.txt, 
> HIVE-1662.12.patch.txt, HIVE-1662.8.patch.txt, HIVE-1662.9.patch.txt, 
> HIVE-1662.D8391.1.patch, HIVE-1662.D8391.2.patch, HIVE-1662.D8391.3.patch, 
> HIVE-1662.D8391.4.patch, HIVE-1662.D8391.5.patch, HIVE-1662.D8391.6.patch, 
> HIVE-1662.D8391.7.patch
>
>
> now hive support filename virtual column. 
> if a file name filter presents in a query, hive should be able to only add 
> files which passed the filter to input paths.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6644) document TestStreaming_2 e2e test case for webhcat

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-6644:


 Summary: document TestStreaming_2 e2e test case for webhcat
 Key: HIVE-6644
 URL: https://issues.apache.org/jira/browse/HIVE-6644
 Project: Hive
  Issue Type: Bug
  Components: WebHCat
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6607) describe extended on a view fails with NPE

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932767#comment-13932767
 ] 

Lefty Leverenz commented on HIVE-6607:
--

Okay, thanks Eugene.

> describe extended on a view fails with NPE
> --
>
> Key: HIVE-6607
> URL: https://issues.apache.org/jira/browse/HIVE-6607
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.14.0
>
> Attachments: HIVE-6607.patch
>
>
> STEPS TO REPRODUCE:
> Create a table called 'sample_08'
> Create a view of the table. From hive command line, please run:
> hive> create view sample_09 as select * from sample_08 ;
> ACTUAL BEHAVIOR:
> Run the following command in the browser:
> http://localhost:50111/templeton/v1/ddl/database/default/table/sample_09?format=extended
> It fails with the following exception:
> {"errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: Exception 
> while processing show table status\n\tat 
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2707)\n\tat
>  org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:343)\n\tat 
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)\n\tat 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)\n\tat
>  org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)\n\tat 
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)\n\tat 
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)\n\tat 
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)\n\tat 
> org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:259)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:213)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:172)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:597)\n\tat 
> org.apache.hadoop.util.RunJar.main(RunJar.java:212)\nCaused by: 
> java.lang.NullPointerException\n\tat 
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.putFileSystemsStats(JsonMetaDataFormatter.java:264)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeOneTableStatus(JsonMetaDataFormatter.java:218)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeAllTableStatus(JsonMetaDataFormatter.java:170)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.showTableStatus(JsonMetaDataFormatter.java:153)\n\tat
>  
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2702)\n\t...
>  16 more\n","error":"FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table 
> status","sqlState":"08S01","errorCode":4,"database":"default","table":"sample_09"}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6607) describe extended on a view fails with NPE

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932763#comment-13932763
 ] 

Lefty Leverenz commented on HIVE-6607:
--

(Didn't notice it's WebHCat.)  Should this bug be documented here?

* [WebHCat Reference GetTable:  Parameters 
|https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+GetTable#WebHCatReferenceGetTable-Parameters]

> describe extended on a view fails with NPE
> --
>
> Key: HIVE-6607
> URL: https://issues.apache.org/jira/browse/HIVE-6607
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.14.0
>
> Attachments: HIVE-6607.patch
>
>
> STEPS TO REPRODUCE:
> Create a table called 'sample_08'
> Create a view of the table. From hive command line, please run:
> hive> create view sample_09 as select * from sample_08 ;
> ACTUAL BEHAVIOR:
> Run the following command in the browser:
> http://localhost:50111/templeton/v1/ddl/database/default/table/sample_09?format=extended
> It fails with the following exception:
> {"errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: Exception 
> while processing show table status\n\tat 
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2707)\n\tat
>  org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:343)\n\tat 
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)\n\tat 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)\n\tat
>  org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)\n\tat 
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)\n\tat 
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)\n\tat 
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)\n\tat 
> org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:259)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:213)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:172)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:597)\n\tat 
> org.apache.hadoop.util.RunJar.main(RunJar.java:212)\nCaused by: 
> java.lang.NullPointerException\n\tat 
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.putFileSystemsStats(JsonMetaDataFormatter.java:264)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeOneTableStatus(JsonMetaDataFormatter.java:218)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeAllTableStatus(JsonMetaDataFormatter.java:170)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.showTableStatus(JsonMetaDataFormatter.java:153)\n\tat
>  
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2702)\n\t...
>  16 more\n","error":"FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table 
> status","sqlState":"08S01","errorCode":4,"database":"default","table":"sample_09"}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6607) describe extended on a view fails with NPE

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932759#comment-13932759
 ] 

Eugene Koifman commented on HIVE-6607:
--

it's a WebHCat bug and I don't think it needs a doc

> describe extended on a view fails with NPE
> --
>
> Key: HIVE-6607
> URL: https://issues.apache.org/jira/browse/HIVE-6607
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.14.0
>
> Attachments: HIVE-6607.patch
>
>
> STEPS TO REPRODUCE:
> Create a table called 'sample_08'
> Create a view of the table. From hive command line, please run:
> hive> create view sample_09 as select * from sample_08 ;
> ACTUAL BEHAVIOR:
> Run the following command in the browser:
> http://localhost:50111/templeton/v1/ddl/database/default/table/sample_09?format=extended
> It fails with the following exception:
> {"errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: Exception 
> while processing show table status\n\tat 
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2707)\n\tat
>  org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:343)\n\tat 
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)\n\tat 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)\n\tat
>  org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)\n\tat 
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)\n\tat 
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)\n\tat 
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)\n\tat 
> org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:259)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:213)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:172)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:597)\n\tat 
> org.apache.hadoop.util.RunJar.main(RunJar.java:212)\nCaused by: 
> java.lang.NullPointerException\n\tat 
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.putFileSystemsStats(JsonMetaDataFormatter.java:264)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeOneTableStatus(JsonMetaDataFormatter.java:218)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeAllTableStatus(JsonMetaDataFormatter.java:170)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.showTableStatus(JsonMetaDataFormatter.java:153)\n\tat
>  
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2702)\n\t...
>  16 more\n","error":"FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table 
> status","sqlState":"08S01","errorCode":4,"database":"default","table":"sample_09"}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6607) describe extended on a view fails with NPE

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932757#comment-13932757
 ] 

Lefty Leverenz commented on HIVE-6607:
--

Should the userdoc mention this bug and when it's going to be fixed?

* [Language Manual DDL:  Describe Table/View/Column 
|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-DescribeTable/View/Column]

> describe extended on a view fails with NPE
> --
>
> Key: HIVE-6607
> URL: https://issues.apache.org/jira/browse/HIVE-6607
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.14.0
>
> Attachments: HIVE-6607.patch
>
>
> STEPS TO REPRODUCE:
> Create a table called 'sample_08'
> Create a view of the table. From hive command line, please run:
> hive> create view sample_09 as select * from sample_08 ;
> ACTUAL BEHAVIOR:
> Run the following command in the browser:
> http://localhost:50111/templeton/v1/ddl/database/default/table/sample_09?format=extended
> It fails with the following exception:
> {"errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: Exception 
> while processing show table status\n\tat 
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2707)\n\tat
>  org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:343)\n\tat 
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)\n\tat 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)\n\tat
>  org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)\n\tat 
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)\n\tat 
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)\n\tat 
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)\n\tat 
> org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:259)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:213)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:172)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:597)\n\tat 
> org.apache.hadoop.util.RunJar.main(RunJar.java:212)\nCaused by: 
> java.lang.NullPointerException\n\tat 
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.putFileSystemsStats(JsonMetaDataFormatter.java:264)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeOneTableStatus(JsonMetaDataFormatter.java:218)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeAllTableStatus(JsonMetaDataFormatter.java:170)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.showTableStatus(JsonMetaDataFormatter.java:153)\n\tat
>  
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2702)\n\t...
>  16 more\n","error":"FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table 
> status","sqlState":"08S01","errorCode":4,"database":"default","table":"sample_09"}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6455:
-

Attachment: HIVE-6455.15.patch

Fixed the test failures caused by ReduceSinkDeDuplication

> Scalable dynamic partitioning and bucketing optimization
> 
>
> Key: HIVE-6455
> URL: https://issues.apache.org/jira/browse/HIVE-6455
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, 
> HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.11.patch, 
> HIVE-6455.12.patch, HIVE-6455.13.patch, HIVE-6455.13.patch, 
> HIVE-6455.14.patch, HIVE-6455.15.patch, HIVE-6455.2.patch, HIVE-6455.3.patch, 
> HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, 
> HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
>
>
> The current implementation of dynamic partition works by keeping at least one 
> record writer open per dynamic partition directory. In case of bucketing 
> there can be multispray file writers which further adds up to the number of 
> open record writers. The record writers of column oriented file format (like 
> ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or 
> compression buffers) open all the time to buffer up the rows and compress 
> them before flushing it to disk. Since these buffers are maintained per 
> column basis the amount of constant memory that will required at runtime 
> increases as the number of partitions and number of columns per partition 
> increases. This often leads to OutOfMemory (OOM) exception in mappers or 
> reducers depending on the number of open record writers. Users often tune the 
> JVM heapsize (runtime memory) to get over such OOM issues. 
> With this optimization, the dynamic partition columns and bucketing columns 
> (in case of bucketed tables) are sorted before being fed to the reducers. 
> Since the partitioning and bucketing columns are sorted, each reducers can 
> keep only one record writer open at any time thereby reducing the memory 
> pressure on the reducers. This optimization is highly scalable as the number 
> of partition and number of columns per partition increases at the cost of 
> sorting the columns.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6620) UDF printf doesn't take either CHAR or VARCHAR as the first argument

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932746#comment-13932746
 ] 

Jason Dere commented on HIVE-6620:
--

+1 if tests pass

> UDF printf doesn't take either CHAR or VARCHAR as the first argument
> 
>
> Key: HIVE-6620
> URL: https://issues.apache.org/jira/browse/HIVE-6620
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-6620.patch
>
>
> {code}
> hive> desc vc;
> OK
> c char(5) None
> vcvarchar(7)  None
> s string  None
> hive> select printf(c) from vc;
> FAILED: SemanticException [Error 10016]: Line 1:14 Argument type mismatch 
> 'c': Argument 1 of function PRINTF must be "string", but "char(5)" was found.
> {code}
> However, if the argument is string type, the query runs successfully.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6457) Ensure Parquet integration has good error messages for data types not supported

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6457:


Fix Version/s: (was: 0.14.0)
   0.13.0

> Ensure Parquet integration has good error messages for data types not 
> supported
> ---
>
> Key: HIVE-6457
> URL: https://issues.apache.org/jira/browse/HIVE-6457
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>  Labels: parquet
> Fix For: 0.13.0
>
> Attachments: HIVE-6457.patch, HIVE-6457.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6457) Ensure Parquet integration has good error messages for data types not supported

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932729#comment-13932729
 ] 

Harish Butani commented on HIVE-6457:
-

committed to 0.13.0 branch

> Ensure Parquet integration has good error messages for data types not 
> supported
> ---
>
> Key: HIVE-6457
> URL: https://issues.apache.org/jira/browse/HIVE-6457
> Project: Hive
>  Issue Type: Improvement
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>  Labels: parquet
> Fix For: 0.13.0
>
> Attachments: HIVE-6457.patch, HIVE-6457.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6575) select * fails on parquet table with map datatype

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6575:


Fix Version/s: (was: 0.14.0)
   0.13.0

> select * fails on parquet table with map datatype
> -
>
> Key: HIVE-6575
> URL: https://issues.apache.org/jira/browse/HIVE-6575
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>  Labels: parquet
> Fix For: 0.13.0
>
> Attachments: HIVE-6575.2.patch, HIVE-6575.3.patch, HIVE-6575.patch
>
>
> Create parquet table with map and run select * from parquet_table, returns 
> following exception:
> {noformat}
>  FAILED: RuntimeException java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.io.parquet.serde.DeepParquetHiveMapInspector cannot 
> be cast to 
> org.apache.hadoop.hive.ql.io.parquet.serde.StandardParquetHiveMapInspector
> {noformat}
> However select  from parquet_table seems to work, and thus joins will 
> work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6575) select * fails on parquet table with map datatype

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932721#comment-13932721
 ] 

Harish Butani commented on HIVE-6575:
-

committed to 0.13.0 branch

> select * fails on parquet table with map datatype
> -
>
> Key: HIVE-6575
> URL: https://issues.apache.org/jira/browse/HIVE-6575
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
>  Labels: parquet
> Fix For: 0.13.0
>
> Attachments: HIVE-6575.2.patch, HIVE-6575.3.patch, HIVE-6575.patch
>
>
> Create parquet table with map and run select * from parquet_table, returns 
> following exception:
> {noformat}
>  FAILED: RuntimeException java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.io.parquet.serde.DeepParquetHiveMapInspector cannot 
> be cast to 
> org.apache.hadoop.hive.ql.io.parquet.serde.StandardParquetHiveMapInspector
> {noformat}
> However select  from parquet_table seems to work, and thus joins will 
> work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6514) TestExecDriver/HCat Pig tests fails with -Phadoop-2

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6514:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk & 0.13. Thanks, Jason!

> TestExecDriver/HCat Pig tests fails with -Phadoop-2
> ---
>
> Key: HIVE-6514
> URL: https://issues.apache.org/jira/browse/HIVE-6514
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 0.13.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.13.0
>
> Attachments: HIVE-6514.1.patch, HIVE-6514.2.patch, HIVE-6514.3.patch
>
>
> Running TestExecDriver with -Phadoop-2 results in the error below. Looks like 
> the test isn't able to access LocalClientProtocolProvider.
> {noformat}
> java.io.IOException: Cannot initialize Cluster. Please check your 
> configuration for mapreduce.framework.name and the correspond server 
> addresses.
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
> at org.apache.hadoop.mapred.JobClient.(JobClient.java:449)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:396)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:739)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.io.IOException(Cannot initialize 
> Cluster. Please check your configuration for mapreduce.framework.name and the 
> correspond server addresses.)'
> Execution failed with exit status: 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932700#comment-13932700
 ] 

Ashutosh Chauhan commented on HIVE-6606:


+1

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.2.patch, HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6639) Vectorization: Partition column names are not picked up.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-6639:
---

Attachment: HIVE-6639.2.patch

> Vectorization: Partition column names are not picked up.
> 
>
> Key: HIVE-6639
> URL: https://issues.apache.org/jira/browse/HIVE-6639
> Project: Hive
>  Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6639.2.patch
>
>
> Vectorization: Partition column names are not picked up causing an NPE.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932695#comment-13932695
 ] 

Ashutosh Chauhan commented on HIVE-6635:


This will double # of threads for HS2 and worse these threads are not coming 
from shared threadpool of HS2. How about putting heartbeat while we are waiting 
for MR job to finish. There is a while loop waiting polling for MR job status. 
I think we can put heartbeating there. This is in 
HadoopJobExecHelper::progress()

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HIVE-6084) WebHCat TestStreaming_2 e2e test should return FAILURE after HIVE-5511

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan resolved HIVE-6084.
-

Resolution: Not A Problem

Closing the jira based on [~ekoifman] 's comments.

> WebHCat TestStreaming_2 e2e test should return FAILURE after HIVE-5511
> --
>
> Key: HIVE-6084
> URL: https://issues.apache.org/jira/browse/HIVE-6084
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-6084.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6606:
-

Attachment: HIVE-6606.2.patch

New version of the patch with changes based on Ashutosh's comments.

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.2.patch, HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6606:
-

Status: Patch Available  (was: Open)

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.2.patch, HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932685#comment-13932685
 ] 

Alan Gates commented on HIVE-6635:
--

[~rhbutani] This patch should go into the 0.13 branch as well.

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932683#comment-13932683
 ] 

Alan Gates commented on HIVE-6606:
--

[~rhbutani] This patch should go into the 0.13 branch as well.

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.2.patch, HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6606:
-

Status: Open  (was: Patch Available)

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6643) Add a check for cross products in plans and output a warning

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932671#comment-13932671
 ] 

Harish Butani commented on HIVE-6643:
-

review at: https://reviews.apache.org/r/19165/

> Add a check for cross products in plans and output a warning
> 
>
> Key: HIVE-6643
> URL: https://issues.apache.org/jira/browse/HIVE-6643
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6643.1.patch
>
>
> Now that we support old style join syntax, it is easy to write queries that 
> generate a plan with a cross product.
> For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and 
> C.z = D.z
> So the JoinTree is:
> A — B
> |__  D — C
> Since we don't reorder join graphs, we will end up with a cross product 
> between (A join B) and C



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 19165: HIVE-6643: Add a check for cross products in plans and output a warning

[Harish Butani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19165/
---

Review request for hive and Gunther Hagleitner.


Bugs: HIVE-6643
https://issues.apache.org/jira/browse/HIVE-6643


Repository: hive-git


Description
---

Now that we support old style join syntax, it is easy to write queries that 
generate a plan with a cross product.
For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z 
= D.z
So the JoinTree is:
A — B
__ D — C
Since we don't reorder join graphs, we will end up with a cross product between 
(A join B) and C


Diffs
-

  itests/qtest/pom.xml f8b81a2 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 
d593d08 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a5e6cbf 
  ql/src/test/queries/clientpositive/cross_product_check_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/cross_product_check_2.q PRE-CREATION 
  ql/src/test/results/clientpositive/cross_product_check_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/cross_product_check_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/19165/diff/


Testing
---

added new tests


Thanks,

Harish Butani

[jira] [Commented] (HIVE-6643) Add a check for cross products in plans and output a warning

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932668#comment-13932668
 ] 

Harish Butani commented on HIVE-6643:
-

preliminary patch attached. There are some issues with this:
- for MapJoins: the warning only shows the bigTable (even this is not 
available). Hopefully the Vertex/Stage and Operator information  is enough for 
the user to see where the cross product is happening.
- for Shuffle Joins: we need the HIVE-4293 changes to show the tables in a 
Reducer that has more than 1 input.
- the .q.out files have the warnings, but these are appearing before the 
preehooks for the queries. Is this ok? 

> Add a check for cross products in plans and output a warning
> 
>
> Key: HIVE-6643
> URL: https://issues.apache.org/jira/browse/HIVE-6643
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6643.1.patch
>
>
> Now that we support old style join syntax, it is easy to write queries that 
> generate a plan with a cross product.
> For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and 
> C.z = D.z
> So the JoinTree is:
> A — B
> |__  D — C
> Since we don't reorder join graphs, we will end up with a cross product 
> between (A join B) and C



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6643) Add a check for cross products in plans and output a warning

[Harish Butani (JIRA)]

Harish Butani created HIVE-6643:
---

 Summary: Add a check for cross products in plans and output a 
warning
 Key: HIVE-6643
 URL: https://issues.apache.org/jira/browse/HIVE-6643
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
 Attachments: HIVE-6643.1.patch

Now that we support old style join syntax, it is easy to write queries that 
generate a plan with a cross product.
For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z 
= D.z
So the JoinTree is:

A — B
|__  D — C

Since we don't reorder join graphs, we will end up with a cross product between 
(A join B) and C



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6643) Add a check for cross products in plans and output a warning

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-6643:


Attachment: HIVE-6643.1.patch

> Add a check for cross products in plans and output a warning
> 
>
> Key: HIVE-6643
> URL: https://issues.apache.org/jira/browse/HIVE-6643
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-6643.1.patch
>
>
> Now that we support old style join syntax, it is easy to write queries that 
> generate a plan with a cross product.
> For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and 
> C.z = D.z
> So the JoinTree is:
> A — B
> |__  D — C
> Since we don't reorder join graphs, we will end up with a cross product 
> between (A join B) and C



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6570) Hive variable substitution does not work with the "source" command

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932644#comment-13932644
 ] 

Ashutosh Chauhan commented on HIVE-6570:


If I am reading the patch correctly, it does that interpretation only for 
{{source}} statement, not for anything else.

> Hive variable substitution does not work with the "source" command
> --
>
> Key: HIVE-6570
> URL: https://issues.apache.org/jira/browse/HIVE-6570
> Project: Hive
>  Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6570.1.patch
>
>
> The following does not work:
> {code}
> source ${hivevar:test-dir}/test.q;
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6514) TestExecDriver/HCat Pig tests fails with -Phadoop-2

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6514:
-

Issue Type: Sub-task  (was: Bug)
Parent: HIVE-6588

> TestExecDriver/HCat Pig tests fails with -Phadoop-2
> ---
>
> Key: HIVE-6514
> URL: https://issues.apache.org/jira/browse/HIVE-6514
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests
>Affects Versions: 0.13.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6514.1.patch, HIVE-6514.2.patch, HIVE-6514.3.patch
>
>
> Running TestExecDriver with -Phadoop-2 results in the error below. Looks like 
> the test isn't able to access LocalClientProtocolProvider.
> {noformat}
> java.io.IOException: Cannot initialize Cluster. Please check your 
> configuration for mapreduce.framework.name and the correspond server 
> addresses.
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
> at org.apache.hadoop.mapred.JobClient.(JobClient.java:449)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:396)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:739)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.io.IOException(Cannot initialize 
> Cluster. Please check your configuration for mapreduce.framework.name and the 
> correspond server addresses.)'
> Execution failed with exit status: 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 19149: Stand alone metastore fails to start if new transaction values not defined in config

[Alan Gates]

> On March 12, 2014, 8:21 p.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java, 
> > line 205
> > 
> >
> > Do we need to synchronize this method?

This is really intended for use only in testing.  It's only in the src area 
rather than test so that it can be picked up cross package for things like 
streaming and hive client tests.  So I'm not too worried about synchronization 
or performance (for the next comment).  I can add comments on the methods to 
make this clear so no one uses it when they shouldn't.


> On March 12, 2014, 8:21 p.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java, 
> > line 215
> > 
> >
> > You created prop object but didn't make use of it. Don't you want to 
> > use that prop here, instead of new Properties?

Oops.  Will fix.


- Alan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19149/#review36974
---


On March 12, 2014, 7:20 p.m., Alan Gates wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19149/
> ---
> 
> (Updated March 12, 2014, 7:20 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-6606
> https://issues.apache.org/jira/browse/HIVE-6606
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The metastore creates instances of TxnHandler. The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.
> 
> Fixed this by changing transaction connection to use the same JDBC connection 
> string as the rest of the metastore.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38 
>   metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java 
> bbb0d28 
>   metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
> 4441c2f 
>   metastore/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 560fd5a 
> 
> Diff: https://reviews.apache.org/r/19149/diff/
> 
> 
> Testing
> ---
> 
> Ran unit tests plus ran on cluster to assure issue not seen when transaction 
> handling turned off.
> 
> 
> Thanks,
> 
> Alan Gates
> 
>

[jira] [Commented] (HIVE-6588) Unit test failures with hadoop-2

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932640#comment-13932640
 ] 

Jason Dere commented on HIVE-6588:
--

Yeah I've noticed a bunch of the following:
1. fix by adding a order by after group by as [~szehon] mentions
2. A bunch of tests only run in hadoop-2 and the diffs have gone stale because 
we only test with hadoop-1.

Was going to use this Jira as an umbrella task for the various fixes.

> Unit test failures with hadoop-2
> 
>
> Key: HIVE-6588
> URL: https://issues.apache.org/jira/browse/HIVE-6588
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>
> A number of tests are failing with -Phadoop-2



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932632#comment-13932632
 ] 

Alan Gates commented on HIVE-6635:
--

Review board: https://reviews.apache.org/r/19161/

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 19161: Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19161/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-6635
https://issues.apache.org/jira/browse/HIVE-6635


Repository: hive-git


Description
---

Added a thread to Driver to send heartbeats. This thread only runs during the 
main loop in Driver.execute. I added this in a separate thread because 
otherwise I would have needed to add threads in every task to see if heartbeats 
needed to be sent. This would be very invasive, and also it's not clear it 
would be possible to cover all cases as there are actions that may simply take 
a long time (like certain metastore operations). The downside is that a query 
will keep running even after it's found out it's locks were aborted and only be 
terminated at the end.


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
4441c2f 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7dbb8be 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java 535912f 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7773f66 

Diff: https://reviews.apache.org/r/19161/diff/


Testing
---

Ran unit tests specific to transaction operations, as well as manual system 
testing.


Thanks,

Alan Gates

[jira] [Commented] (HIVE-6588) Unit test failures with hadoop-2

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932627#comment-13932627
 ] 

Szehon Ho commented on HIVE-6588:
-

Typo:  "fixed by adding a order by after group by"

> Unit test failures with hadoop-2
> 
>
> Key: HIVE-6588
> URL: https://issues.apache.org/jira/browse/HIVE-6588
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>
> A number of tests are failing with -Phadoop-2



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6635:
-

Attachment: HIVE-6635.patch

Added a thread to Driver to send heartbeats.  This thread only runs during the 
main loop in Driver.execute.  I added this in a separate thread because 
otherwise I would have needed to add threads in every task to see if heartbeats 
needed to be sent.  This would be very invasive, and also it's not clear it 
would be possible to cover all cases as there are actions that may simply take 
a long time (like certain metastore operations).  The downside is that a query 
will keep running even after it's found out it's locks were aborted and only be 
terminated at the end.

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6635) Heartbeats are not being sent when DbLockMgr is used and an operation holds locks

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-6635:
-

Status: Patch Available  (was: Open)

> Heartbeats are not being sent when DbLockMgr is used and an operation holds 
> locks
> -
>
> Key: HIVE-6635
> URL: https://issues.apache.org/jira/browse/HIVE-6635
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.13.0
>
> Attachments: HIVE-6635.patch
>
>
> The new DbLockManager depends on heartbeats from the client in order to 
> determine that a lock has not timed out.  The client is not currently sending 
> those heartbeats.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6588) Unit test failures with hadoop-2

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932624#comment-13932624
 ] 

Szehon Ho commented on HIVE-6588:
-

Just wanted to share my experience, a lot of these are caused by MAPREDUCE-434 
which is in hadoop 2.3.0, hive MR jobs in test env had been running only one 
reducer in local mode, now suddenly there are more.  

I think the majority of these can be fixed by adding a group-by, or having a 
different version of the test for hadoop1,2 versions for others, like those 
asserting number of buckets.  Will try to fix some if I get some cycles.

> Unit test failures with hadoop-2
> 
>
> Key: HIVE-6588
> URL: https://issues.apache.org/jira/browse/HIVE-6588
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>
> A number of tests are failing with -Phadoop-2



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6514) TestExecDriver/HCat Pig tests fails with -Phadoop-2

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932622#comment-13932622
 ] 

Ashutosh Chauhan commented on HIVE-6514:


+1

> TestExecDriver/HCat Pig tests fails with -Phadoop-2
> ---
>
> Key: HIVE-6514
> URL: https://issues.apache.org/jira/browse/HIVE-6514
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.13.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6514.1.patch, HIVE-6514.2.patch, HIVE-6514.3.patch
>
>
> Running TestExecDriver with -Phadoop-2 results in the error below. Looks like 
> the test isn't able to access LocalClientProtocolProvider.
> {noformat}
> java.io.IOException: Cannot initialize Cluster. Please check your 
> configuration for mapreduce.framework.name and the correspond server 
> addresses.
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75)
> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
> at org.apache.hadoop.mapred.JobClient.(JobClient.java:449)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:396)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:739)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.io.IOException(Cannot initialize 
> Cluster. Please check your configuration for mapreduce.framework.name and the 
> correspond server addresses.)'
> Execution failed with exit status: 1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6607) describe extended on a view fails with NPE

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6607:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Eugene!

> describe extended on a view fails with NPE
> --
>
> Key: HIVE-6607
> URL: https://issues.apache.org/jira/browse/HIVE-6607
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.14.0
>
> Attachments: HIVE-6607.patch
>
>
> STEPS TO REPRODUCE:
> Create a table called 'sample_08'
> Create a view of the table. From hive command line, please run:
> hive> create view sample_09 as select * from sample_08 ;
> ACTUAL BEHAVIOR:
> Run the following command in the browser:
> http://localhost:50111/templeton/v1/ddl/database/default/table/sample_09?format=extended
> It fails with the following exception:
> {"errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: Exception 
> while processing show table status\n\tat 
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2707)\n\tat
>  org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:343)\n\tat 
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)\n\tat 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)\n\tat
>  org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)\n\tat 
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)\n\tat 
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)\n\tat 
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)\n\tat 
> org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:259)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:213)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:172)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:597)\n\tat 
> org.apache.hadoop.util.RunJar.main(RunJar.java:212)\nCaused by: 
> java.lang.NullPointerException\n\tat 
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.putFileSystemsStats(JsonMetaDataFormatter.java:264)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeOneTableStatus(JsonMetaDataFormatter.java:218)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeAllTableStatus(JsonMetaDataFormatter.java:170)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.showTableStatus(JsonMetaDataFormatter.java:153)\n\tat
>  
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2702)\n\t...
>  16 more\n","error":"FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table 
> status","sqlState":"08S01","errorCode":4,"database":"default","table":"sample_09"}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6569) HCatalog still has references to deprecated property hive.metastore.local

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6569:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Sushanth!

> HCatalog still has references to deprecated property hive.metastore.local
> -
>
> Key: HIVE-6569
> URL: https://issues.apache.org/jira/browse/HIVE-6569
> Project: Hive
>  Issue Type: Bug
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Minor
>  Labels: cleanup, hcatalog
> Fix For: 0.14.0
>
> Attachments: HIVE-6569.2.patch, HIVE-6569.patch
>
>
> HIVE-2585 removed the conf parameter hive.metastore.local, but HCatalog still 
> has references to it. Most of it is in tests, but one is in PigHCatUtil, 
> which leads to HCatLoader/HCatStorer jobs giving warnings. We need to remove 
> them.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6607) describe extended on a view fails with NPE

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932600#comment-13932600
 ] 

Hive QA commented on HIVE-6607:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633789/HIVE-6607.patch

{color:green}SUCCESS:{color} +1 5387 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1715/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1715/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12633789

> describe extended on a view fails with NPE
> --
>
> Key: HIVE-6607
> URL: https://issues.apache.org/jira/browse/HIVE-6607
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-6607.patch
>
>
> STEPS TO REPRODUCE:
> Create a table called 'sample_08'
> Create a view of the table. From hive command line, please run:
> hive> create view sample_09 as select * from sample_08 ;
> ACTUAL BEHAVIOR:
> Run the following command in the browser:
> http://localhost:50111/templeton/v1/ddl/database/default/table/sample_09?format=extended
> It fails with the following exception:
> {"errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: Exception 
> while processing show table status\n\tat 
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2707)\n\tat
>  org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:343)\n\tat 
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)\n\tat 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)\n\tat
>  org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)\n\tat 
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)\n\tat 
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)\n\tat 
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)\n\tat 
> org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:259)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:213)\n\tat 
> org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:172)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:597)\n\tat 
> org.apache.hadoop.util.RunJar.main(RunJar.java:212)\nCaused by: 
> java.lang.NullPointerException\n\tat 
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.putFileSystemsStats(JsonMetaDataFormatter.java:264)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeOneTableStatus(JsonMetaDataFormatter.java:218)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.makeAllTableStatus(JsonMetaDataFormatter.java:170)\n\tat
>  
> org.apache.hadoop.hive.ql.metadata.formatting.JsonMetaDataFormatter.showTableStatus(JsonMetaDataFormatter.java:153)\n\tat
>  
> org.apache.hadoop.hive.ql.exec.DDLTask.showTableStatus(DDLTask.java:2702)\n\t...
>  16 more\n","error":"FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing show table 
> status","sqlState":"08S01","errorCode":4,"database":"default","table":"sample_09"}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6613) Control when spcific Inputs / Outputs are started

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932593#comment-13932593
 ] 

Gunther Hagleitner commented on HIVE-6613:
--

Looks good. Cool to see this - should really help with memory in reducers.

Couple comments:

- Can you avoid creating a conf in TezCacheAccess? Maybe just pass it in get().
- Have you considered adding the input to the cache key instead of using a Set? 
That way you can also remove the lock (I'm assuming the ObjectRegistry handles 
that).
- You can drop the getLocalWork check in the tez hashtable loader. Tez doesn't 
have local work.
- The javadoc of the init function needs to be updated with your changes.

> Control when spcific Inputs / Outputs are started
> -
>
> Key: HIVE-6613
> URL: https://issues.apache.org/jira/browse/HIVE-6613
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: TEZ-6613.1.txt
>
>
> When running with Tez - a couple of enhancement are possible
> 1) Avoid re-fetching data in case of MapJoins - since the data is likely to 
> be cached after the first run (container re-use for the same query)
> 2) Start Outputs only after required Inputs are ready - specifically useful 
> in case of Reduce - where shuffle requires a large memory, and the Output (if 
> it's a sorted output) also requires a fair amount of memory.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6572) Use shimmed version of hadoop conf names for mapred.{min,max}.split.size{.*}

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6572:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk & 0.13 Thanks, Sushanth!

> Use shimmed version of hadoop conf names for mapred.{min,max}.split.size{.*}
> 
>
> Key: HIVE-6572
> URL: https://issues.apache.org/jira/browse/HIVE-6572
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 0.13.0
>
> Attachments: HIVE-6572.patch
>
>
> HadoopShims has a method to fetch config parameters by name so that they 
> return the appropriate config param name for the appropriate hadoop version. 
> We need to be consistent about using these versions.
> For eg:. mapred.min.split.size is deprecated with hadoop 2.x, and is instead 
> called mapreduce.input.fileinputformat.split.minsize .
> Also, there is a bug in Hadoop23Shims, Hadoop20SShims and Hadoop20Shims that 
> defines MAPREDMINSPLITSIZEPERNODE as mapred.min.split.size.per.rack and 
> MAPREDMINSPLITSIZEPERRACK as mapred.min.split.size.per.node. This is wrong 
> and confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6543) TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync is failing sometimes

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932517#comment-13932517
 ] 

Ashutosh Chauhan commented on HIVE-6543:


I think this doesnt require full test run. There is no code change. Only test 
change. So, if test pass, we are good. I will commit this shortly, unless 
[~xuefuz] thinks otherwise.

> TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync is failing 
> sometimes
> -
>
> Key: HIVE-6543
> URL: https://issues.apache.org/jira/browse/HIVE-6543
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-6543.1.patch.txt
>
>
> NO PRECOMMIT TESTS
> The test uses "CREATE TABLE NON_EXISTING_TAB (ID STRING) location 
> 'hdfs://localhost:1/a/b/c'" query for intended fail but it seemed not 
> fail so quickly in testbed. Just making the query worse (replacing hdfs to 
> invalid, etc.) would be enough.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6642) Query fails to vectorize when a non string partition column is part of the query expression

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6642:
---

 Summary: Query fails to vectorize when a non string partition 
column is part of the query expression
 Key: HIVE-6642
 URL: https://issues.apache.org/jira/browse/HIVE-6642
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


drop table if exists alltypesorc_part;

CREATE TABLE alltypesorc_part (
ctinyint tinyint,
csmallint smallint,
cint int,
cbigint bigint,
cfloat float,
cdouble double,
cstring1 string,
cstring2 string,
ctimestamp1 timestamp,
ctimestamp2 timestamp,
cboolean1 boolean,
cboolean2 boolean) partitioned by (ds int) STORED AS ORC;

insert overwrite table alltypesorc_part partition (ds=2011) select * from 
alltypesorc limit 100;
insert overwrite table alltypesorc_part partition (ds=2012) select * from 
alltypesorc limit 200;

explain select *
from (select ds from alltypesorc_part) t1,
 alltypesorc t2
where t1.ds = t2.cint
order by t2.ctimestamp1
limit 100;

The above query fails to vectorize because (select ds from alltypesorc_part) t1 
returns a string column and the join equality on t2 is performed on an int 
column. The correct output when vectorization is turned on should be:
STAGE DEPENDENCIES:
  Stage-5 is a root stage
  Stage-2 depends on stages: Stage-5
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-5
Map Reduce Local Work
  Alias -> Map Local Tables:
t1:alltypesorc_part
  Fetch Operator
limit: -1
  Alias -> Map Local Operator Tree:
t1:alltypesorc_part
  TableScan
alias: alltypesorc_part
Statistics: Num rows: 300 Data size: 62328 Basic stats: COMPLETE 
Column stats: COMPLETE
Select Operator
  expressions: ds (type: int)
  outputColumnNames: _col0
  Statistics: Num rows: 300 Data size: 1200 Basic stats: COMPLETE 
Column stats: COMPLETE
  HashTable Sink Operator
condition expressions:
  0 {_col0}
  1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
{cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
keys:
  0 _col0 (type: int)
  1 cint (type: int)

  Stage: Stage-2
Map Reduce
  Map Operator Tree:
  TableScan
alias: t2
Statistics: Num rows: 3536 Data size: 1131711 Basic stats: COMPLETE 
Column stats: NONE
Map Join Operator
  condition map:
   Inner Join 0 to 1
  condition expressions:
0 {_col0}
1 {ctinyint} {csmallint} {cint} {cbigint} {cfloat} {cdouble} 
{cstring1} {cstring2} {ctimestamp1} {ctimestamp2} {cboolean1} {cboolean2}
  keys:
0 _col0 (type: int)
1 cint (type: int)
  outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
_col6, _col7, _col8, _col9, _col10, _col11, _col12
  Statistics: Num rows: 3889 Data size: 1244882 Basic stats: 
COMPLETE Column stats: NONE
  Filter Operator
predicate: (_col0 = _col3) (type: boolean)
Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
COMPLETE Column stats: NONE
Select Operator
  expressions: _col0 (type: int), _col1 (type: tinyint), _col2 
(type: smallint), _col3 (type: int), _col4 (type: bigint), _col5 (type: float), 
_col6 (type: double), _col7 (type: string), _col8 (type: string), _col\
9 (type: timestamp), _col10 (type: timestamp), _col11 (type: boolean), _col12 
(type: boolean)
  outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, 
_col6, _col7, _col8, _col9, _col10, _col11, _col12
  Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
COMPLETE Column stats: NONE
  Reduce Output Operator
key expressions: _col9 (type: timestamp)
sort order: +
Statistics: Num rows: 1944 Data size: 622280 Basic stats: 
COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: 
tinyint), _col2 (type: smallint), _col3 (type: int), _col4 (type: bigint), 
_col5 (type: float), _col6 (type: double), _col7 (type: string), _col8 (type: 
strin\
g), _col9 (type: timestamp), _col10 (type: timestamp), _col11 (type: boolean), 
_col12 (type: boolean)
  Local Work:
Map Reduce Local Work
  Execution mode: vectorized
  Reduce Operator Tree:
Extract
  Statistics: Num rows: 1944 Data size: 622280 Basic stats: COMPLETE 
Column stats: NONE
  Limit
Number of rows: 100
Statistics: Num rows: 100 Data size: 32000 Basic stats: COMPLETE 
Column

[jira] [Commented] (HIVE-6569) HCatalog still has references to deprecated property hive.metastore.local

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932509#comment-13932509
 ] 

Ashutosh Chauhan commented on HIVE-6569:


+1

> HCatalog still has references to deprecated property hive.metastore.local
> -
>
> Key: HIVE-6569
> URL: https://issues.apache.org/jira/browse/HIVE-6569
> Project: Hive
>  Issue Type: Bug
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Minor
>  Labels: cleanup, hcatalog
> Attachments: HIVE-6569.2.patch, HIVE-6569.patch
>
>
> HIVE-2585 removed the conf parameter hive.metastore.local, but HCatalog still 
> has references to it. Most of it is in tests, but one is in PigHCatUtil, 
> which leads to HCatLoader/HCatStorer jobs giving warnings. We need to remove 
> them.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6570) Hive variable substitution does not work with the "source" command

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932506#comment-13932506
 ] 

Edward Capriolo commented on HIVE-6570:
---

WE should make a release note if someone has  $ in there file hive might now 
try to interpret it.

> Hive variable substitution does not work with the "source" command
> --
>
> Key: HIVE-6570
> URL: https://issues.apache.org/jira/browse/HIVE-6570
> Project: Hive
>  Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6570.1.patch
>
>
> The following does not work:
> {code}
> source ${hivevar:test-dir}/test.q;
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6570) Hive variable substitution does not work with the "source" command

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932501#comment-13932501
 ] 

Ashutosh Chauhan commented on HIVE-6570:


yeah, failure is unrelated. I will commit this shortly.

> Hive variable substitution does not work with the "source" command
> --
>
> Key: HIVE-6570
> URL: https://issues.apache.org/jira/browse/HIVE-6570
> Project: Hive
>  Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6570.1.patch
>
>
> The following does not work:
> {code}
> source ${hivevar:test-dir}/test.q;
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6570) Hive variable substitution does not work with the "source" command

[Anthony Hsu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932494#comment-13932494
 ] 

Anthony Hsu commented on HIVE-6570:
---

The single test failure seems unrelated to my change.

Ashutosh, can you commit my change for me?

> Hive variable substitution does not work with the "source" command
> --
>
> Key: HIVE-6570
> URL: https://issues.apache.org/jira/browse/HIVE-6570
> Project: Hive
>  Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6570.1.patch
>
>
> The following does not work:
> {code}
> source ${hivevar:test-dir}/test.q;
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6084) WebHCat TestStreaming_2 e2e test should return FAILURE after HIVE-5511

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932462#comment-13932462
 ] 

Eugene Koifman commented on HIVE-6084:
--


This test is supposed to have check_job_complete=SUCCESS. This field is a 
property of the Launcher task which is either ends in SUCCESS(3) or KILLED(5). 
(short of bugs/unexpected failures). It is not the property of the task being 
submitted. check_job_exit_value is the exit value for subprocess.
Thus since this test specifies a mapper which returns 1 (i.e. no-zero status), 
MR fails it. So exit_value=1 and job_complete=SUCCESS is correct.

> WebHCat TestStreaming_2 e2e test should return FAILURE after HIVE-5511
> --
>
> Key: HIVE-6084
> URL: https://issues.apache.org/jira/browse/HIVE-6084
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-6084.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6641) optimized HashMap keys won't work correctly with decimals

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6641:
--

 Summary: optimized HashMap keys won't work correctly with decimals
 Key: HIVE-6641
 URL: https://issues.apache.org/jira/browse/HIVE-6641
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Decimal values with can be equal while having different byte representations 
(different precision/scale), so comparing bytes is not enough. For a quick fix, 
we can disable this for decimals



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6447) Bucket map joins in hive-tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932388#comment-13932388
 ] 

Hive QA commented on HIVE-6447:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633855/HIVE-6447.2.patch

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 5387 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testBuildDag
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testClose
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testEmptyWork
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hadoop.hive.ql.plan.TestTezWork.testBroadcastConnect
org.apache.hadoop.hive.ql.plan.TestTezWork.testConnect
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1714/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1714/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12633855

> Bucket map joins in hive-tez
> 
>
> Key: HIVE-6447
> URL: https://issues.apache.org/jira/browse/HIVE-6447
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6447.1.patch, HIVE-6447.2.patch, HIVE-6447.WIP.patch
>
>
> Support bucket map joins in tez.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6611) Joining multiple union all outputs fails on Tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6611:
-

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks Vikram!

> Joining multiple union all outputs fails on Tez
> ---
>
> Key: HIVE-6611
> URL: https://issues.apache.org/jira/browse/HIVE-6611
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6611.1.patch
>
>
> Queries like:
> with u as (select * from src union all select * from src)
> select * from u join u;
> will fail on Tez because only one union flows into the join reduce phase.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6620) UDF printf doesn't take either CHAR or VARCHAR as the first argument

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6620:
--

Attachment: HIVE-6620.patch

> UDF printf doesn't take either CHAR or VARCHAR as the first argument
> 
>
> Key: HIVE-6620
> URL: https://issues.apache.org/jira/browse/HIVE-6620
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-6620.patch
>
>
> {code}
> hive> desc vc;
> OK
> c char(5) None
> vcvarchar(7)  None
> s string  None
> hive> select printf(c) from vc;
> FAILED: SemanticException [Error 10016]: Line 1:14 Argument type mismatch 
> 'c': Argument 1 of function PRINTF must be "string", but "char(5)" was found.
> {code}
> However, if the argument is string type, the query runs successfully.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6620) UDF printf doesn't take either CHAR or VARCHAR as the first argument

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6620:
--

Status: Patch Available  (was: Open)

> UDF printf doesn't take either CHAR or VARCHAR as the first argument
> 
>
> Key: HIVE-6620
> URL: https://issues.apache.org/jira/browse/HIVE-6620
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-6620.patch
>
>
> {code}
> hive> desc vc;
> OK
> c char(5) None
> vcvarchar(7)  None
> s string  None
> hive> select printf(c) from vc;
> FAILED: SemanticException [Error 10016]: Line 1:14 Argument type mismatch 
> 'c': Argument 1 of function PRINTF must be "string", but "char(5)" was found.
> {code}
> However, if the argument is string type, the query runs successfully.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6640) Change hive.version.shortname in hive 0.13 branch to '0.13.0'

[Harish Butani (JIRA)]

Harish Butani created HIVE-6640:
---

 Summary: Change hive.version.shortname in hive 0.13 branch to 
'0.13.0'
 Key: HIVE-6640
 URL: https://issues.apache.org/jira/browse/HIVE-6640
 Project: Hive
  Issue Type: Bug
Reporter: Harish Butani
Assignee: Harish Butani
Priority: Trivial
 Fix For: 0.13.0


[~jdere] thanks for pointing this out.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-3364) LOAD DATA LOCAL INPATH : Misleading error message on Permission problem

[Chandra Sekhar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chandra Sekhar updated HIVE-3364:
-

Affects Version/s: (was: 0.12.0)
   Issue Type: Bug  (was: Improvement)

> LOAD DATA LOCAL INPATH : Misleading error message on Permission problem
> ---
>
> Key: HIVE-3364
> URL: https://issues.apache.org/jira/browse/HIVE-3364
> Project: Hive
>  Issue Type: Bug
>Reporter: Xavier Gillmann
>Priority: Trivial
>
> When launching a LOAD DATA LOCAL INPATH '/path/to/file' ... command
> for which hive user has no access rights.
> Error message is "FAILED: Error in semantic analysis: Line 1:23 Invalid 
> path..."
> Could be interesting to bubble up the "Permission denied"



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-3364) LOAD DATA LOCAL INPATH : Misleading error message on Permission problem

[Chandra Sekhar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chandra Sekhar updated HIVE-3364:
-

Affects Version/s: 0.12.0
   Status: Patch Available  (was: Open)

> LOAD DATA LOCAL INPATH : Misleading error message on Permission problem
> ---
>
> Key: HIVE-3364
> URL: https://issues.apache.org/jira/browse/HIVE-3364
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.12.0
>Reporter: Xavier Gillmann
>Priority: Trivial
>
> When launching a LOAD DATA LOCAL INPATH '/path/to/file' ... command
> for which hive user has no access rights.
> Error message is "FAILED: Error in semantic analysis: Line 1:23 Invalid 
> path..."
> Could be interesting to bubble up the "Permission denied"



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6570) Hive variable substitution does not work with the "source" command

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932339#comment-13932339
 ] 

Ashutosh Chauhan commented on HIVE-6570:


+1

> Hive variable substitution does not work with the "source" command
> --
>
> Key: HIVE-6570
> URL: https://issues.apache.org/jira/browse/HIVE-6570
> Project: Hive
>  Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6570.1.patch
>
>
> The following does not work:
> {code}
> source ${hivevar:test-dir}/test.q;
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6538) yet another annoying exception in test logs

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6538:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

committed to trunk

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Szehon Ho
>Priority: Trivial
> Fix For: 0.14.0
>
> Attachments: HIVE-6538.2.patch, HIVE-6538.2.patch, HIVE-6538.patch
>
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit

[jira] [Commented] (HIVE-6572) Use shimmed version of hadoop conf names for mapred.{min,max}.split.size{.*}

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932317#comment-13932317
 ] 

Ashutosh Chauhan commented on HIVE-6572:


+1

> Use shimmed version of hadoop conf names for mapred.{min,max}.split.size{.*}
> 
>
> Key: HIVE-6572
> URL: https://issues.apache.org/jira/browse/HIVE-6572
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-6572.patch
>
>
> HadoopShims has a method to fetch config parameters by name so that they 
> return the appropriate config param name for the appropriate hadoop version. 
> We need to be consistent about using these versions.
> For eg:. mapred.min.split.size is deprecated with hadoop 2.x, and is instead 
> called mapreduce.input.fileinputformat.split.minsize .
> Also, there is a bug in Hadoop23Shims, Hadoop20SShims and Hadoop20Shims that 
> defines MAPREDMINSPLITSIZEPERNODE as mapred.min.split.size.per.rack and 
> MAPREDMINSPLITSIZEPERRACK as mapred.min.split.size.per.node. This is wrong 
> and confusing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6636) /user/hive is a bad default for HDFS jars path for Tez

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6636:
---

Attachment: HIVE-6636.02.patch

i think it works the other way around, overriding the default default; anyway I 
can change it

> /user/hive is a bad default for HDFS jars path for Tez
> --
>
> Key: HIVE-6636
> URL: https://issues.apache.org/jira/browse/HIVE-6636
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6636.01.patch, HIVE-6636.02.patch, HIVE-6636.patch
>
>
> If user runs hive under the user name that is not "hive", jobs will fail 
> until everyone is granted write access to /user/hive, which is not nice.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6634) Add test case for HIVE-6575

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-6634:


Attachment: HIVE-6634.patch

FYI [~xuefuz], added some statements that fails without the patch, succeeds 
with.

> Add test case for HIVE-6575
> ---
>
> Key: HIVE-6634
> URL: https://issues.apache.org/jira/browse/HIVE-6634
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-6634.patch
>
>
> Follow up JIRA to HIVE-6575



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6634) Add test case for HIVE-6575

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-6634:


Status: Patch Available  (was: Open)

> Add test case for HIVE-6575
> ---
>
> Key: HIVE-6634
> URL: https://issues.apache.org/jira/browse/HIVE-6634
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-6634.patch
>
>
> Follow up JIRA to HIVE-6575



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932297#comment-13932297
 ] 

Ashutosh Chauhan commented on HIVE-6606:


Left some comments on RB.

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 19149: Stand alone metastore fails to start if new transaction values not defined in config

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19149/#review36974
---



common/src/java/org/apache/hadoop/hive/conf/HiveConf.java


Good we are deleting this. Previously I wondered why we need this when we 
gonna write to same metastore.



metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java


Do we need to synchronize this method?



metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java


Doing new HiveConf(), should be avoided if possible because:
* Its an expensive operation, perf wise.
* Other conf object in memory may have different values because of set 
commands.
If you aren't able to get hold of hiveconf object, probably we need to make 
changes upstream to pass it all the way down here.




metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java


You created prop object but didn't make use of it. Don't you want to use 
that prop here, instead of new Properties?


- Ashutosh Chauhan


On March 12, 2014, 7:20 p.m., Alan Gates wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19149/
> ---
> 
> (Updated March 12, 2014, 7:20 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-6606
> https://issues.apache.org/jira/browse/HIVE-6606
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The metastore creates instances of TxnHandler. The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.
> 
> Fixed this by changing transaction connection to use the same JDBC connection 
> string as the rest of the metastore.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38 
>   metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java 
> bbb0d28 
>   metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java 
> 4441c2f 
>   metastore/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 560fd5a 
> 
> Diff: https://reviews.apache.org/r/19149/diff/
> 
> 
> Testing
> ---
> 
> Ran unit tests plus ran on cluster to assure issue not seen when transaction 
> handling turned off.
> 
> 
> Thanks,
> 
> Alan Gates
> 
>

[jira] [Created] (HIVE-6639) Vectorization: Partition column names are not picked up.

[Jitendra Nath Pandey (JIRA)]

Jitendra Nath Pandey created HIVE-6639:
--

 Summary: Vectorization: Partition column names are not picked up.
 Key: HIVE-6639
 URL: https://issues.apache.org/jira/browse/HIVE-6639
 Project: Hive
  Issue Type: Bug
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey


Vectorization: Partition column names are not picked up causing an NPE.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6568) Vectorized cast of decimal to string and timestamp produces incorrect result.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-6568:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

> Vectorized cast of decimal to string and timestamp produces incorrect result.
> -
>
> Key: HIVE-6568
> URL: https://issues.apache.org/jira/browse/HIVE-6568
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.13.0
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Fix For: 0.13.0
>
> Attachments: HIVE-6568.1.patch, HIVE-6568.2.patch, HIVE-6568.3.patch
>
>
> A decimal value 1.23 with scale 5 is represented in string as 1.23000. This 
> behavior is different from HiveDecimal behavior.
> The difference in cast to timestamp is due to more aggressive rounding in 
> vectorized expression.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Timeline for the Hive 0.13 release?

[Harish Butani]

added 6457, 6575
@Vaibhav I meant deferring HIVE-6580, since there is no patch for it.
So your list is:
6512, 6625, 5155 and 6625, correct?
On Mar 12, 2014, at 12:15 PM, Xuefu Zhang  wrote:

> One more, HIVE-6457, is also nice to have in 0.13. Could you please include
> it, Harish?
> 
> Thanks,
> Xuefu
> 
> 
> On Wed, Mar 12, 2014 at 10:53 AM, Xuefu Zhang  wrote:
> 
>> Hi Harish,
>> 
>> Could you please add HIVE-6575, which is already resolved in trunk, but
>> bad enough to be fixed for 0.13 as well?
>> 
>> Thanks,
>> Xuefu
>> 
>> 
>> On Wed, Mar 12, 2014 at 3:13 AM, Vaibhav Gumashta <
>> vgumas...@hortonworks.com> wrote:
>> 
>>> Sure, we can defer 6068. Actually 6602 too, since there are a couple of
>>> related jiras that need to be addressed. However, if possible can we have
>>> https://issues.apache.org/jira/browse/HIVE-6625 in 13? Its big brother
>>> https://issues.apache.org/jira/browse/HIVE-5155 is ready to be committed
>>> and HIVE-6625 is a minor patch on top of it.
>>> 
>>> Thanks a lot!
>>> 
>>> 
>>> On Tue, Mar 11, 2014 at 1:17 PM, Harish Butani >>> wrote:
>>> 
 ok with the first 3.
 HIVE-6068 doesn't have a patch yet. Can this be deferred?
 
 On Mar 11, 2014, at 12:06 PM, Vaibhav Gumashta <
>>> vgumas...@hortonworks.com>
 wrote:
 
> Can you please consider the following:
> https://issues.apache.org/jira/browse/HIVE-6602 (committed to trunk),
> https://issues.apache.org/jira/browse/HIVE-6512,
> https://issues.apache.org/jira/browse/HIVE-6068,
> https://issues.apache.org/jira/browse/HIVE-6580. Most of them are bug
 fixes.
> 
> Thanks,
> --Vaibhav
> 
> 
> On Tue, Mar 11, 2014 at 8:39 AM, Harish Butani <
>>> hbut...@hortonworks.com
> wrote:
> 
>> yes sure.
>> 
>> 
>> On Mar 10, 2014, at 3:55 PM, Gopal V  wrote:
>> 
>>> Can I add HIVE-6518 as well to the merge queue on
>>> 
>>> 
>> 
 
>>> https://cwiki.apache.org/confluence/display/Hive/Hive+0.13+release+status
>>> 
>>> It is a relatively simple OOM safety patch to vectorized group-by.
>>> 
>>> Tests pass locally for vec group-by, but the pre-commit tests
>>> haven't
>> fired eventhough it's been PA for a while now.
>>> 
>>> Cheers,
>>> Gopal
>> 
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or
 entity to
>> which it is addressed and may contain information that is
>>> confidential,
>> privileged and exempt from disclosure under applicable law. If the
 reader
>> of this message is not the intended recipient, you are hereby
>>> notified
 that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
 immediately
>> and delete it from your system. Thank You.
>> 
> 
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or
>>> entity
 to
> which it is addressed and may contain information that is
>>> confidential,
> privileged and exempt from disclosure under applicable law. If the
>>> reader
> of this message is not the intended recipient, you are hereby notified
 that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender
 immediately
> and delete it from your system. Thank You.
 
 
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
>>> entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
>>> reader
 of this message is not the intended recipient, you are hereby notified
>>> that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
>>> immediately
 and delete it from your system. Thank You.
 
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>> 
>> 
>> 


-- 
CONFIDENTIALITY NOTICE

[jira] [Commented] (HIVE-4764) Support Kerberos HTTP authentication for HiveServer2 running in http mode

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932261#comment-13932261
 ] 

Vaibhav Gumashta commented on HIVE-4764:


[~thejas] Sure, will attach a new one later today.

> Support Kerberos HTTP authentication for HiveServer2 running in http mode
> -
>
> Key: HIVE-4764
> URL: https://issues.apache.org/jira/browse/HIVE-4764
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-4764.1.patch, HIVE-4764.2.patch, HIVE-4764.3.patch, 
> HIVE-4764.4.patch
>
>
> Support Kerberos authentication for HiveServer2 running in http mode.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6568) Vectorized cast of decimal to string and timestamp produces incorrect result.

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932257#comment-13932257
 ] 

Harish Butani commented on HIVE-6568:
-

+1 for 0.13

> Vectorized cast of decimal to string and timestamp produces incorrect result.
> -
>
> Key: HIVE-6568
> URL: https://issues.apache.org/jira/browse/HIVE-6568
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 0.13.0
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6568.1.patch, HIVE-6568.2.patch, HIVE-6568.3.patch
>
>
> A decimal value 1.23 with scale 5 is represented in string as 1.23000. This 
> behavior is different from HiveDecimal behavior.
> The difference in cast to timestamp is due to more aggressive rounding in 
> vectorized expression.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6629) Change SET ROLE NONE to SET ROLE ALL

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-6629:
---

Assignee: Brock Noland
  Status: Patch Available  (was: Open)

> Change SET ROLE NONE to SET ROLE ALL
> 
>
> Key: HIVE-6629
> URL: https://issues.apache.org/jira/browse/HIVE-6629
> Project: Hive
>  Issue Type: Task
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Blocker
> Fix For: 0.13.0
>
> Attachments: HIVE-6629.patch
>
>
> I understand this is well after it's been implemented. However, I think we 
> need to discuss the counter-intuitive SET ROLE semantics before we release 
> 0.13.
> The new Hive Authz work implements "SET ROLE NONE" similar to MySQL meaning 
> that "SET ROLE NONE" actually sets your role to the default. This is 
> extremely counter-intuitive.
> * 
> [Oracle|http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_10004.htm]
>  
> * 
> [Informix|http://pic.dhe.ibm.com/infocenter/idshelp/v117/index.jsp?topic=%2Fcom.ibm.sqls.doc%2Fids_sqs_1186.htm]
> * [Vertica|https://my.vertica.com/docs/5.1.6/HTML/index.htm#15645.htm]
> * 
> [SAP|http://www.sapdb.org/htmhelp/44/a17998442911d3a98200a0c9449261/content.htm]
> All of the widely deployed databases above have "SET ROLE NONE" disable all 
> privileges. Those databases have the intuitive "SET ROLE ALL" or "SET ROLE 
> DEFAULT" enable all privileges.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6538) yet another annoying exception in test logs

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932241#comment-13932241
 ] 

Hive QA commented on HIVE-6538:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633205/HIVE-6538.2.patch

{color:green}SUCCESS:{color} +1 5381 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1712/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1712/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12633205

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Szehon Ho
>Priority: Trivial
> Attachments: HIVE-6538.2.patch, HIVE-6538.2.patch, HIVE-6538.patch
>
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.j

[jira] [Commented] (HIVE-6606) Stand alone metastore fails to start if new transaction values not defined in config

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932230#comment-13932230
 ] 

Alan Gates commented on HIVE-6606:
--

https://reviews.apache.org/r/19149/

> Stand alone metastore fails to start if new transaction values not defined in 
> config
> 
>
> Key: HIVE-6606
> URL: https://issues.apache.org/jira/browse/HIVE-6606
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.13.0
>
> Attachments: HIVE-6606.patch
>
>
> The metastore creates instances of TxnHandler.  The constructor of this class 
> will fail if the config value for the jdbc string it expects is not defined 
> in the config file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)