date:20140106


[ 
https://issues.apache.org/jira/browse/HIVE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862918#comment-13862918
 ] 

Vaibhav Gumashta commented on HIVE-3746:


[~navis] Thanks so much for the patch! I was also curious if there was any 
performance instrumentation done on your side?

 Fix HS2 ResultSet Serialization Performance Regression
 --

 Key: HIVE-3746
 URL: https://issues.apache.org/jira/browse/HIVE-3746
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, Server Infrastructure
Reporter: Carl Steinbach
Assignee: Navis
  Labels: HiveServer2, jdbc, thrift
 Fix For: 0.13.0

 Attachments: HIVE-3746.1.patch.txt, HIVE-3746.2.patch.txt, 
 HIVE-3746.3.patch.txt, HIVE-3746.4.patch.txt, HIVE-3746.5.patch.txt, 
 HIVE-3746.6.patch.txt, HIVE-3746.7.patch.txt






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-3936) Remote debug failed with hadoop 0.23X, hadoop 2.X


 [ 
https://issues.apache.org/jira/browse/HIVE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-3936:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Thank you Swarnim for the contribution! I have committed this to trunk. Thank 
you Thejas for the review and Xie for the report!

 Remote debug failed with hadoop 0.23X, hadoop 2.X
 -

 Key: HIVE-3936
 URL: https://issues.apache.org/jira/browse/HIVE-3936
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.8.0, 0.8.1, 0.9.0
Reporter: Xie Long
Assignee: Swarnim Kulkarni
Priority: Minor
 Fix For: 0.13.0

 Attachments: HIVE-3936.1.patch


 In $HIVE_HOME/bin/hive and $HADOOP_HOME/bin/hadoop, $HADOOP_CLIENT_OPTS is  
 appended to $HADOOP_OPTS, which leads to the problem.
 hive --debug
 ERROR: Cannot load this JVM TI agent twice, check your java command line for 
 duplicate jdwp options.
 Error occurred during initialization of VM
 agent library failed to init: jdwp



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler


[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863009#comment-13863009
 ] 

Brock Noland commented on HIVE-2599:


Great to hear, it sounds like the composite key is working as desired -- both 
inserts and selects work as expected!

The patch looks good to me as well!  The only change required I noticed is that 
the two new classes related to the composite keys require apache license 
headers. I can commit this after that change!

 Support Composit/Compound Keys with HBaseStorageHandler
 ---

 Key: HIVE-2599
 URL: https://issues.apache.org/jira/browse/HIVE-2599
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, 
 HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt


 It would be really nice for hive to be able to understand composite keys from 
 an underlying HBase schema. Currently we have to store key fields twice to be 
 able to both key and make data available. I noticed John Sichi mentioned in 
 HIVE-1228 that this would be a separate issue but I cant find any follow up. 
 How feasible is this in the HBaseStorageHandler?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6145) Change Precommit builds to Java 7

Brock Noland created HIVE-6145:
--

 Summary: Change Precommit builds to Java 7
 Key: HIVE-6145
 URL: https://issues.apache.org/jira/browse/HIVE-6145
 Project: Hive
  Issue Type: Task
Reporter: Brock Noland
Assignee: Brock Noland






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Hive-trunk-hadoop2 - Build # 651 - Still Failing

Changes for Build #640

Changes for Build #641
[navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis 
reviewed by Thejas M Nair)

[navis] HIVE-4257 : java.sql.SQLNonTransientConnectionException on 
JDBCStatsAggregator (Teddy Choi via Navis, reviewed by Ashutosh)


Changes for Build #642

Changes for Build #643
[ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package 
from Microsoft to Hive (Hideaki Kumura via Eric Hanson)


Changes for Build #644
[cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming 
conventions (Sergey Shelukhin via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II 
(Navis via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis 
via cws)

[jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure 
vectorization produces same results as non-vectorized execution (Sergey 
Shelukhin via Jitendra Pandey)


Changes for Build #645

Changes for Build #646
[ehans] HIVE-5757: Implement vectorized support for CASE (Eric Hanson)


Changes for Build #647
[thejas] HIVE-5795 : Hive should be able to skip header and footer rows when 
reading data file for a table (Shuaishuai Nie via Thejas Nair)


Changes for Build #648
[thejas] HIVE-5923 : SQL std auth - parser changes (Thejas Nair, reviewed by 
Brock Noland)


Changes for Build #649

Changes for Build #650

Changes for Build #651
[brock] HIVE-3936 - Remote debug failed with hadoop 0.23X, hadoop 2.X (Swarnim 
Kulkarni via Brock)




No tests ran.

The Apache Jenkins build system has built Hive-trunk-hadoop2 (build #651)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-hadoop2/651/ 
to view the results.

[jira] [Created] (HIVE-6146) Ensure all tests still pass with Java 7

Brock Noland created HIVE-6146:
--

 Summary: Ensure all tests still pass with Java 7
 Key: HIVE-6146
 URL: https://issues.apache.org/jira/browse/HIVE-6146
 Project: Hive
  Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Hive-trunk-h0.21 - Build # 2551 - Still Failing

Changes for Build #2539

Changes for Build #2540
[navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis 
reviewed by Thejas M Nair)


Changes for Build #2541

Changes for Build #2542
[ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package 
from Microsoft to Hive (Hideaki Kumura via Eric Hanson)


Changes for Build #2543
[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II 
(Navis via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis 
via cws)

[jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure 
vectorization produces same results as non-vectorized execution (Sergey 
Shelukhin via Jitendra Pandey)


Changes for Build #2544
[cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming 
conventions (Sergey Shelukhin via cws)


Changes for Build #2545

Changes for Build #2546
[ehans] HIVE-5757: Implement vectorized support for CASE (Eric Hanson)


Changes for Build #2547
[thejas] HIVE-5795 : Hive should be able to skip header and footer rows when 
reading data file for a table (Shuaishuai Nie via Thejas Nair)


Changes for Build #2548
[thejas] HIVE-5923 : SQL std auth - parser changes (Thejas Nair, reviewed by 
Brock Noland)


Changes for Build #2549

Changes for Build #2550

Changes for Build #2551
[brock] HIVE-3936 - Remote debug failed with hadoop 0.23X, hadoop 2.X (Swarnim 
Kulkarni via Brock)




No tests ran.

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2551)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2551/ to 
view the results.

[jira] [Commented] (HIVE-5945) ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.

2014-01-06 Thread Yin Huai (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863047#comment-13863047
 ] 

Yin Huai commented on HIVE-5945:


Thanks Navis for the change. date_dim is a native table. Actually, I think the 
problem is 
org.apache.hadoop.hive.ql.plan.ConditionalResolverCommonJoin.getParticipants. 
It uses ctx.getAliasToTask(); to get all aliases. However, these aliases do not 
include aliases appearing in the MapLocalWork (those small tables.). So for a 
query like 
{code}
set hive.auto.convert.join.noconditionaltask=false;
select
   i_item_id
FROM store_sales
JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
limit 10;
{code}

The plan is 
{code}
STAGE DEPENDENCIES:
  Stage-5 is a root stage , consists of Stage-6, Stage-1
  Stage-6 has a backup stage: Stage-1
  Stage-3 depends on stages: Stage-6
  Stage-1
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-5
Conditional Operator

  Stage: Stage-6
Map Reduce Local Work
  Alias - Map Local Tables:
item 
  Fetch Operator
limit: -1
  Alias - Map Local Operator Tree:
item 
  TableScan
alias: item
HashTable Sink Operator
  condition expressions:
0 
1 {i_item_id}
  handleSkewJoin: false
  keys:
0 [Column[ss_item_sk]]
1 [Column[i_item_sk]]
  Position of Big Table: 0

  Stage: Stage-3
Map Reduce
  Alias - Map Operator Tree:
store_sales 
  TableScan
alias: store_sales
Map Join Operator
  condition map:
   Inner Join 0 to 1
  condition expressions:
0 
1 {i_item_id}
  handleSkewJoin: false
  keys:
0 [Column[ss_item_sk]]
1 [Column[i_item_sk]]
  outputColumnNames: _col26
  Position of Big Table: 0
  Select Operator
expressions:
  expr: _col26
  type: string
outputColumnNames: _col0
Limit
  File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
  Local Work:
Map Reduce Local Work

  Stage: Stage-1
Map Reduce
  Alias - Map Operator Tree:
item 
  TableScan
alias: item
Reduce Output Operator
  key expressions:
expr: i_item_sk
type: int
  sort order: +
  Map-reduce partition columns:
expr: i_item_sk
type: int
  tag: 1
  value expressions:
expr: i_item_id
type: string
store_sales 
  TableScan
alias: store_sales
Reduce Output Operator
  key expressions:
expr: ss_item_sk
type: int
  sort order: +
  Map-reduce partition columns:
expr: ss_item_sk
type: int
  tag: 0
  Reduce Operator Tree:
Join Operator
  condition map:
   Inner Join 0 to 1
  condition expressions:
0 
1 {VALUE._col1}
  handleSkewJoin: false
  outputColumnNames: _col26
  Select Operator
expressions:
  expr: _col26
  type: string
outputColumnNames: _col0
Limit
  File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
Fetch Operator
  limit: 10
{code}
The alias of item will not be in the set returned by getParticipants. Thus, 
the input of sumOfExcept will be 
{code}
aliasToSize: {store_sales=388445409, item=5051899}
aliases: [store_sales]
except: store_sales
{code}
and then we get 0 for the size of small tables.

I think in getParticipants, we can check the type of a task and if it is a 
MapRedTask, we can use getWork().getMapWork().getMapLocalWork() to get the 
local task. Then, we can get aliases of those small tables through aliasToWork.

Another minor comment. Can you add a comment

[jira] [Updated] (HIVE-4515) select count(*) from table query on hive-0.10.0, hbase-0.94.7 integration throws exceptions


 [ 
https://issues.apache.org/jira/browse/HIVE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-4515:
---

Description: 
After integration hive-0.10.0+hbase-0.94.7, these commands could be executed 
successfully:
{noformat}
create table
insert overwrite table
select * from table

However, when execute select count(*) from table, throws exception:
hive select count(*) from test; 
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=number
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=number
In order to set a constant number of reducers:
  set mapred.reduce.tasks=number
Starting Job = job_201305061042_0028, Tracking URL = 
http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028
Kill Command = /opt/modules/hadoop/hadoop-1.0.4/libexec/../bin/hadoop job  
-kill job_201305061042_0028
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2013-05-07 18:41:42,649 Stage-1 map = 0%,  reduce = 0%
2013-05-07 18:42:14,789 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201305061042_0028 with errors
Error during job, obtaining debugging information...
Job Tracking URL: 
http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028
Examining task ID: task_201305061042_0028_m_02 (and more) from job 
job_201305061042_0028

Task with the most failures(4): 
-
Task ID:
  task_201305061042_0028_m_00

URL:
  
http://master0:50030/taskdetails.jsp?jobid=job_201305061042_0028tipid=task_201305061042_0028_m_00
-
Diagnostic Messages for this Task:
java.lang.NegativeArraySizeException: -1
at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148)
at 
org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133)
at 
org.apache.hadoop.hive.hbase.HBaseSplit.readFields(HBaseSplit.java:53)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.readFields(HiveInputFormat.java:150)
at 
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at 
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:396)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)


FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched: 
Job 0: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

==
The log of tasktracker:

stderr logs

13/05/07 18:43:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/05/07 18:43:20 INFO mapred.TaskRunner: Creating symlink: 
/tmp/hadoop-hadoop/mapred/local/taskTracker/distcache/107328478296390_-1298160740_2123690974/master0/tmp/hive-hadoop/hive_2013-05-07_18-41-30_290_832140779606816147/-mr-10003/fd22448b-e923-498c-bc00-2164ca68447d
 - 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/HIVE_PLANfd22448b-e923-498c-bc00-2164ca68447d
13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating 
symlink: 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/jars/javolution
 - 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/javolution
13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating 
symlink: 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/jars/org
 - 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/org
13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating 
symlink: 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/jars/hive-exec-log4j.properties
 - 
/tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/hive-exec-log4j.properties
13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating 
symlink:

Hive-trunk-hadoop2 - Build # 652 - Still Failing

Changes for Build #640

Changes for Build #641
[navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis 
reviewed by Thejas M Nair)

[navis] HIVE-4257 : java.sql.SQLNonTransientConnectionException on 
JDBCStatsAggregator (Teddy Choi via Navis, reviewed by Ashutosh)


Changes for Build #642

Changes for Build #643
[ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package 
from Microsoft to Hive (Hideaki Kumura via Eric Hanson)


Changes for Build #644
[cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming 
conventions (Sergey Shelukhin via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II 
(Navis via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis 
via cws)

[jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure 
vectorization produces same results as non-vectorized execution (Sergey 
Shelukhin via Jitendra Pandey)


Changes for Build #645

Changes for Build #646
[ehans] HIVE-5757: Implement vectorized support for CASE (Eric Hanson)


Changes for Build #647
[thejas] HIVE-5795 : Hive should be able to skip header and footer rows when 
reading data file for a table (Shuaishuai Nie via Thejas Nair)


Changes for Build #648
[thejas] HIVE-5923 : SQL std auth - parser changes (Thejas Nair, reviewed by 
Brock Noland)


Changes for Build #649

Changes for Build #650

Changes for Build #651
[brock] HIVE-3936 - Remote debug failed with hadoop 0.23X, hadoop 2.X (Swarnim 
Kulkarni via Brock)


Changes for Build #652



No tests ran.

The Apache Jenkins build system has built Hive-trunk-hadoop2 (build #652)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-hadoop2/652/ 
to view the results.

[jira] [Commented] (HIVE-6140) trim udf is very slow


[ 
https://issues.apache.org/jira/browse/HIVE-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863101#comment-13863101
 ] 

Eric Hanson commented on HIVE-6140:
---

This may not be relevant for you, but if you can use ORC then you can enable 
vectorized execution, and benefit from the vectorized implementation of TRIM, 
which should be much faster. See 
org.apache.hadoop.hive.ql.exec.vector.expressions.StringTrim.

 trim udf is very slow
 -

 Key: HIVE-6140
 URL: https://issues.apache.org/jira/browse/HIVE-6140
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Thejas M Nair
Assignee: Anandha L Ranganathan
 Attachments: temp.pl


 Paraphrasing what was reported by [~cartershanklin] -
 I used the attached Perl script to generate 500 million two-character strings 
 which always included a space. I loaded it using:
 create table letters (l string); 
 load data local inpath '/home/sandbox/data.csv' overwrite into table letters;
 Then I ran this SQL script:
 select count(l) from letters where l = 'l ';
 select count(l) from letters where trim(l) = 'l';
 First query = 170 seconds
 Second query  = 514 seconds



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-4996) unbalanced calls to openTransaction/commitTransaction

2014-01-06 Thread Sergey Shelukhin (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863105#comment-13863105
 ] 

Sergey Shelukhin commented on HIVE-4996:


HIVE-4807 changed it to bonecp due to another bug. Maybe some additional 
instrumentation (such as tracking db queries) can uncover the cause of this 
bug, instead of just changing it

 unbalanced calls to openTransaction/commitTransaction
 -

 Key: HIVE-4996
 URL: https://issues.apache.org/jira/browse/HIVE-4996
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.10.0, 0.11.0, 0.12.0
 Environment: hiveserver1  Java HotSpot(TM) 64-Bit Server VM (build 
 20.6-b01, mixed mode)
Reporter: wangfeng
Priority: Critical
  Labels: hive, metastore
 Attachments: hive-4996.path

   Original Estimate: 504h
  Remaining Estimate: 504h

 when we used hiveserver1 based on hive-0.10.0, we found the Exception 
 thrown.It was:
 FAILED: Error in metadata: MetaException(message:java.lang.RuntimeException: 
 commitTransaction was called but openTransactionCalls = 0. This probably 
 indicates that the
 re are unbalanced calls to openTransaction/commitTransaction)
 FAILED: Execution Error, return code 1 from 
 org.apache.hadoop.hive.ql.exec.DDLTask
 help



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Carl Steinbach

The Apache Hive PMC has voted to make Vikram Dixit a committer on the
Apache Hive Project.

Please join me in congratulating Vikram!

Thanks.

Carl

Hive-trunk-h0.21 - Build # 2552 - Still Failing

Changes for Build #2539

Changes for Build #2540
[navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis 
reviewed by Thejas M Nair)


Changes for Build #2541

Changes for Build #2542
[ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package 
from Microsoft to Hive (Hideaki Kumura via Eric Hanson)


Changes for Build #2543
[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II 
(Navis via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis 
via cws)

[jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure 
vectorization produces same results as non-vectorized execution (Sergey 
Shelukhin via Jitendra Pandey)


Changes for Build #2544
[cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming 
conventions (Sergey Shelukhin via cws)


Changes for Build #2545

Changes for Build #2546
[ehans] HIVE-5757: Implement vectorized support for CASE (Eric Hanson)


Changes for Build #2547
[thejas] HIVE-5795 : Hive should be able to skip header and footer rows when 
reading data file for a table (Shuaishuai Nie via Thejas Nair)


Changes for Build #2548
[thejas] HIVE-5923 : SQL std auth - parser changes (Thejas Nair, reviewed by 
Brock Noland)


Changes for Build #2549

Changes for Build #2550

Changes for Build #2551
[brock] HIVE-3936 - Remote debug failed with hadoop 0.23X, hadoop 2.X (Swarnim 
Kulkarni via Brock)


Changes for Build #2552



No tests ran.

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2552)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2552/ to 
view the results.

Re: Review Request 16007: HIVE-5946 - DDL authorization task factory should be pluggable and tested

2014-01-06 Thread Brock Noland



 On Jan. 2, 2014, 9:52 p.m., Thejas Nair wrote:
  ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestDefaultHiveAuthorizationTaskFactory.java,
   line 430
  https://reviews.apache.org/r/16007/diff/2/?file=393936#file393936line430
 
  A nit - 
  I found it a little unintuitive to return the list from assertSize() 
  call. Should we just have return type as void and do the check on another 
  line.
  I don't have a strong opinion about this, so keeping it this way is 
  also fine.
  
 

I created a tiny class to match this fluent without adding a bunch of extra LOC.


- Brock


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16007/#review31067
---


On Jan. 6, 2014, 4:31 p.m., Brock Noland wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/16007/
 ---
 
 (Updated Jan. 6, 2014, 4:31 p.m.)
 
 
 Review request for hive.
 
 
 Bugs: HIVE-5946
 https://issues.apache.org/jira/browse/HIVE-5946
 
 
 Repository: hive-git
 
 
 Description
 ---
 
 Moves the task creation logic to a factory which is now tested with 20 unit 
 tests.
 
 
 Diffs
 -
 
   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1f242ea 
   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d0807d2 
   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
 448dae2 
   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
 8bd0927 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactory.java
  PRE-CREATION 
   
 ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
  PRE-CREATION 
   
 ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
  PRE-CREATION 
 
 Diff: https://reviews.apache.org/r/16007/diff/
 
 
 Testing
 ---
 
 20 additional unit tests which pass
 
 
 Thanks,
 
 Brock Noland

Re: Review Request 16007: HIVE-5946 - DDL authorization task factory should be pluggable and tested

2014-01-06 Thread Brock Noland


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16007/
---

(Updated Jan. 6, 2014, 4:31 p.m.)


Review request for hive.


Changes
---

Latest patch.


Bugs: HIVE-5946
https://issues.apache.org/jira/browse/HIVE-5946


Repository: hive-git


Description
---

Moves the task creation logic to a factory which is now tested with 20 unit 
tests.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1f242ea 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d0807d2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 448dae2 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 8bd0927 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactory.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
 PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/16007/diff/


Testing
---

20 additional unit tests which pass


Thanks,

Brock Noland

[jira] [Updated] (HIVE-5946) DDL authorization task factory should be better tested


 [ 
https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5946:
---

Attachment: HIVE-5946.patch

New patch modified to remove the pluggable aspect and rebased on trunk.

 DDL authorization task factory should be better tested
 --

 Key: HIVE-5946
 URL: https://issues.apache.org/jira/browse/HIVE-5946
 Project: Hive
  Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-5946.patch, HIVE-5946.patch


 Thejas is working on various authorization issues and one element that might 
 be useful in that effort and increase test coverage and testability would be 
 perform authorization task creation in a factory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Sergey Shelukhin

Congrats Vikram!


On Mon, Jan 6, 2014 at 8:58 AM, Carl Steinbach c...@apache.org wrote:

 The Apache Hive PMC has voted to make Vikram Dixit a committer on the
 Apache Hive Project.

 Please join me in congratulating Vikram!

 Thanks.

 Carl


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Jarek Jarcec Cecho

Congratulations Vikram!

Jarcec

On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
 The Apache Hive PMC has voted to make Vikram Dixit a committer on the
 Apache Hive Project.
 
 Please join me in congratulating Vikram!
 
 Thanks.
 
 Carl


signature.asc
Description: Digital signature

[jira] [Commented] (HIVE-6100) Introduce basic set operations as UDFs

2014-01-06 Thread Alan Gates (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863121#comment-13863121
 ] 

Alan Gates commented on HIVE-6100:
--

These would all be valuable additions to Hive's SQL.  Why not add them into the 
language directly rather than as UDFs?

 Introduce basic set operations as UDFs
 --

 Key: HIVE-6100
 URL: https://issues.apache.org/jira/browse/HIVE-6100
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Kostiantyn Kudriavtsev
Priority: Minor
 Fix For: 0.13.0


 Introduce basic set operations:
 1. Intersection: The intersection of A and B, denoted by A ∩ B, is the set of 
 all things that are members of both A and B.
 select set_intersection(arr_a, arr_b) from dual
 2. Union: The union of A and B, denoted by A ∪ B, is the set of all things 
 that are members of either A or B.
 select set_union(arr_a, arr_b) from dual
 3. Symmetric difference: the symmetric difference of two sets is the set of 
 elements which are in either of the sets and not in their intersection.
 select set_symdiff(arr_a, arr_b) from dual



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler

2014-01-06 Thread Nick Dimiduk (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863122#comment-13863122
 ] 

Nick Dimiduk commented on HIVE-2599:


After a cursory review, patch v3 looks good to me also. It should be trivial to 
extend these concepts to provide a LazyStructObjectInspector over the 
Strict/StructIterator types in HBase. Let me see about finding time to do so 
this week.

 Support Composit/Compound Keys with HBaseStorageHandler
 ---

 Key: HIVE-2599
 URL: https://issues.apache.org/jira/browse/HIVE-2599
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, 
 HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt


 It would be really nice for hive to be able to understand composite keys from 
 an underlying HBase schema. Currently we have to store key fields twice to be 
 able to both key and make data available. I noticed John Sichi mentioned in 
 HIVE-1228 that this would be a separate issue but I cant find any follow up. 
 How feasible is this in the HBaseStorageHandler?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Thejas Nair

Congrats Vikram!


On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho jar...@apache.org wrote:
 Congratulations Vikram!

 Jarcec

 On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
 The Apache Hive PMC has voted to make Vikram Dixit a committer on the
 Apache Hive Project.

 Please join me in congratulating Vikram!

 Thanks.

 Carl

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Hari Subramaniyan

congrats Vikram!!




On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com wrote:

 Congrats Vikram!


 On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho jar...@apache.org
 wrote:
  Congratulations Vikram!
 
  Jarcec
 
  On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
  The Apache Hive PMC has voted to make Vikram Dixit a committer on the
  Apache Hive Project.
 
  Please join me in congratulating Vikram!
 
  Thanks.
 
  Carl

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Gunther Hagleitner

Congratulations Vikram!

Thanks,
Gunther.


On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan 
hsubramani...@hortonworks.com wrote:

 congrats Vikram!!




 On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com
 wrote:

  Congrats Vikram!
 
 
  On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho jar...@apache.org
  wrote:
   Congratulations Vikram!
  
   Jarcec
  
   On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
   The Apache Hive PMC has voted to make Vikram Dixit a committer on the
   Apache Hive Project.
  
   Please join me in congratulating Vikram!
  
   Thanks.
  
   Carl
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-6100) Introduce basic set operations as UDFs

2014-01-06 Thread Edward Capriolo (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863147#comment-13863147
 ] 

Edward Capriolo commented on HIVE-6100:
---

Having UDFs would still be useful. I use a lot of nested structures. We end up 
doing really complicated and kinda slow lateral view / join queries to do set 
operations sometimes. Having UDFs that did things on complex types could help 
in many situations.

 Introduce basic set operations as UDFs
 --

 Key: HIVE-6100
 URL: https://issues.apache.org/jira/browse/HIVE-6100
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Kostiantyn Kudriavtsev
Priority: Minor
 Fix For: 0.13.0


 Introduce basic set operations:
 1. Intersection: The intersection of A and B, denoted by A ∩ B, is the set of 
 all things that are members of both A and B.
 select set_intersection(arr_a, arr_b) from dual
 2. Union: The union of A and B, denoted by A ∪ B, is the set of all things 
 that are members of either A or B.
 select set_union(arr_a, arr_b) from dual
 3. Symmetric difference: the symmetric difference of two sets is the set of 
 elements which are in either of the sets and not in their intersection.
 select set_symdiff(arr_a, arr_b) from dual



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Eugene Koifman

Congratulations!


On Mon, Jan 6, 2014 at 9:44 AM, Gunther Hagleitner 
ghagleit...@hortonworks.com wrote:

 Congratulations Vikram!

 Thanks,
 Gunther.


 On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan 
 hsubramani...@hortonworks.com wrote:

 congrats Vikram!!




 On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com
 wrote:

  Congrats Vikram!
 
 
  On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho jar...@apache.org
  wrote:
   Congratulations Vikram!
  
   Jarcec
  
   On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
   The Apache Hive PMC has voted to make Vikram Dixit a committer on the
   Apache Hive Project.
  
   Please join me in congratulating Vikram!
  
   Thanks.
  
   Carl
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.



 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Updated] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler

2014-01-06 Thread Swarnim Kulkarni (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swarnim Kulkarni updated HIVE-2599:
---

Attachment: HIVE-2599.4.patch.txt

Updated patch with apache license headers.

 Support Composit/Compound Keys with HBaseStorageHandler
 ---

 Key: HIVE-2599
 URL: https://issues.apache.org/jira/browse/HIVE-2599
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, 
 HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt


 It would be really nice for hive to be able to understand composite keys from 
 an underlying HBase schema. Currently we have to store key fields twice to be 
 able to both key and make data available. I noticed John Sichi mentioned in 
 HIVE-1228 that this would be a separate issue but I cant find any follow up. 
 How feasible is this in the HBaseStorageHandler?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5946) DDL authorization task factory should be better tested

2014-01-06 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863214#comment-13863214
 ] 

Hive QA commented on HIVE-5946:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621618/HIVE-5946.patch

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 4895 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantGroupTable
org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantRoleTable
org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testGrantUserTable
org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeGroupTable
org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeRoleTable
org.apache.hadoop.hive.ql.parse.authorization.TestHiveAuthorizationTaskFactory.testRevokeUserTable
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/811/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/811/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12621618

 DDL authorization task factory should be better tested
 --

 Key: HIVE-5946
 URL: https://issues.apache.org/jira/browse/HIVE-5946
 Project: Hive
  Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-5946.patch, HIVE-5946.patch


 Thejas is working on various authorization issues and one element that might 
 be useful in that effort and increase test coverage and testability would be 
 perform authorization task creation in a factory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

2014-01-06 Thread Xuefu Zhang (JIRA)

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863231#comment-13863231
]

Xuefu Zhang commented on HIVE-6131:
---

Can you please clarify in your step #6 whether you're editing local file or
HDFS file? If you're editing your local file, then you will need to load the
data again as you do in step #3, or manually replace the file on HDFS with your
local edited file.

New columns after table alter result in null values despite data

Key: HIVE-6131
URL: https://issues.apache.org/jira/browse/HIVE-6131
Project: Hive
Issue Type: Bug
Reporter: James Vaughan
Priority: Minor

Hi folks,
I found and verified a bug on our CDH 4.0.3 install of Hive when adding
columns to tables with Partitions using 'REPLACE COLUMNS'. I dug through the
Jira a little bit and didn't see anything for it so hopefully this isn't just
noise on the radar.
Basically, when you alter a table with partitions and then reupload data to
that partition, it doesn't seem to recognize the extra data that actually
exists in HDFS- as in, returns NULL values on the new column despite having
the data and recognizing the new column in the metadata.
Here's some steps to reproduce using a basic table:
1. Run this hive command: CREATE TABLE jvaughan_test (col1 string)
partitioned by (day string);
2. Create a simple file on the system with a couple of entries, something
like hi and hi2 separated by newlines.
3. Run this hive command, pointing it at the file: LOAD DATA LOCAL INPATH
'FILEDIR' OVERWRITE INTO TABLE jvaughan_test PARTITION (day = '2014-01-02');
4. Confirm the data with: SELECT * FROM jvaughan_test WHERE day =
'2014-01-02';
5. Alter the column definitions: ALTER TABLE jvaughan_test REPLACE COLUMNS
(col1 string, col2 string);
6. Edit your file and add a second column using the default separator
(ctrl+v, then ctrl+a in Vim) and add two more entries, such as hi3 on the
first row and hi4 on the second
7. Run step 3 again
8. Check the data again like in step 4
For me, this is the results that get returned:
hive select * from jvaughan_test where day = '2014-01-01';
OK
hiNULL2014-01-02
hi2 NULL2014-01-02
This is despite the fact that there is data in the file stored by the
partition in HDFS.
Let me know if you need any other information. The only workaround for me
currently is to drop partitions for any I'm replacing data in and THEN
reupload the new data file.
Thanks,
-James

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6131) New columns after table alter result in null values despite data

2014-01-06 Thread James Vaughan (JIRA)

[
https://issues.apache.org/jira/browse/HIVE-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863242#comment-13863242
]

James Vaughan commented on HIVE-6131:
-

Editing the local file. Step 7 is supposed to handle re-uploading the file
using an OVERWRITE command like you say.

New columns after table alter result in null values despite data

Key: HIVE-6131
URL: https://issues.apache.org/jira/browse/HIVE-6131
Project: Hive
Issue Type: Bug
Reporter: James Vaughan
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6130) Webhcat resource /templeton/v1/status doesn't track Hive job progress

2014-01-06 Thread Shuaishuai Nie (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shuaishuai Nie updated HIVE-6130:
-

Attachment: HIVE-6130.1.patch

Hi [~ekoifman]. Yes, but we still need the part in the original patch witch 
parse the hive job id correctly. Attached the re-based patch.

 Webhcat resource /templeton/v1/status doesn't track Hive job progress
 -

 Key: HIVE-6130
 URL: https://issues.apache.org/jira/browse/HIVE-6130
 Project: Hive
  Issue Type: Bug
Reporter: Shuaishuai Nie
 Attachments: HIVE-6130.1.patch


 Problem: Unable to track Hive job progress through webhcat.
 Cause:
 TempletonUtils has code to parse PIG and JAR child jobid and percentage 
 progress but none for Hive jobs. extractPercentComplete() and 
 extractChildJobId() are supposed to do this parsing. 
 Effect:
 /templeton/v1/queue/$job_id?user.name=$user returns no job progress info. The 
 jobid param in the json result that's supposed to contain the child hive 
 jobid contains the templetoncontrollerjob id instead leaving the parent jobid 
 null.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6003) bin/hive --debug should not append HIVE_CLIENT_OPTS to HADOOP_OPTS


 [ 
https://issues.apache.org/jira/browse/HIVE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6003:


Resolution: Duplicate
Status: Resolved  (was: Patch Available)

Fixed in HIVE-3936

 bin/hive --debug should not append HIVE_CLIENT_OPTS to HADOOP_OPTS 
 ---

 Key: HIVE-6003
 URL: https://issues.apache.org/jira/browse/HIVE-6003
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-6003.1.patch


 hadoop (0.20.2, 1.x, 2.x) appends HADOOP_CLIENT_OPTS to HADOO_OPTS. 
 So it is unnecessary to have this statement in bin/hive, under debug mode -
  
   export HADOOP_OPTS=$HADOOP_OPTS $HADOOP_CLIENT_OPTS
 It results in the HADOOP_CLIENT_OPTS being appended twice, resulting in this 
 error in debug mode.
 {code}
 bin/hive --debug 
 ERROR: Cannot load this JVM TI agent twice, check your java command line for 
 duplicate jdwp options.
 Error occurred during initialization of VM
 agent library failed to init: jdwp
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Prasanth Jayachandran

Congratulations Vikram!!

Thanks
Prasanth Jayachandran

On Jan 6, 2014, at 11:50 PM, Eugene Koifman ekoif...@hortonworks.com wrote:

 Congratulations!
 
 
 On Mon, Jan 6, 2014 at 9:44 AM, Gunther Hagleitner 
 ghagleit...@hortonworks.com wrote:
 
 Congratulations Vikram!
 
 Thanks,
 Gunther.
 
 
 On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan 
 hsubramani...@hortonworks.com wrote:
 
 congrats Vikram!!
 
 
 
 
 On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com
 wrote:
 
 Congrats Vikram!
 
 
 On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho jar...@apache.org
 wrote:
 Congratulations Vikram!
 
 Jarcec
 
 On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
 The Apache Hive PMC has voted to make Vikram Dixit a committer on the
 Apache Hive Project.
 
 Please join me in congratulating Vikram!
 
 Thanks.
 
 Carl
 
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
 entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
 reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.
 
 
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.
 
 
 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.
 
 
 -- 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to 
 which it is addressed and may contain information that is confidential, 
 privileged and exempt from disclosure under applicable law. If the reader 
 of this message is not the intended recipient, you are hereby notified that 
 any printing, copying, dissemination, distribution, disclosure or 
 forwarding of this communication is strictly prohibited. If you have 
 received this communication in error, please contact the sender immediately 
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-6100) Introduce basic set operations as UDFs


[ 
https://issues.apache.org/jira/browse/HIVE-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863262#comment-13863262
 ] 

Kostiantyn Kudriavtsev commented on HIVE-6100:
--

Alan, thank you for your comment
Could you please added Hive's SQL example, because your proposal is not totally 
clear for me? 

 Introduce basic set operations as UDFs
 --

 Key: HIVE-6100
 URL: https://issues.apache.org/jira/browse/HIVE-6100
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Kostiantyn Kudriavtsev
Priority: Minor
 Fix For: 0.13.0


 Introduce basic set operations:
 1. Intersection: The intersection of A and B, denoted by A ∩ B, is the set of 
 all things that are members of both A and B.
 select set_intersection(arr_a, arr_b) from dual
 2. Union: The union of A and B, denoted by A ∪ B, is the set of all things 
 that are members of either A or B.
 select set_union(arr_a, arr_b) from dual
 3. Symmetric difference: the symmetric difference of two sets is the set of 
 elements which are in either of the sets and not in their intersection.
 select set_symdiff(arr_a, arr_b) from dual



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6147) Support avro data stored in HBase columns

2014-01-06 Thread Swarnim Kulkarni (JIRA)

Swarnim Kulkarni created HIVE-6147:
--

 Summary: Support avro data stored in HBase columns
 Key: HIVE-6147
 URL: https://issues.apache.org/jira/browse/HIVE-6147
 Project: Hive
  Issue Type: Bug
  Components: HBase Handler
Affects Versions: 0.12.0
Reporter: Swarnim Kulkarni


Presently, the HBase Hive integration supports querying only primitive data 
types in columns. It would be nice to be able to store and query Avro objects 
in HBase columns by making them visible as structs to Hive. This will allow 
Hive to perform ad hoc analysis of HBase data which can be deeply structured.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6046) add UDF for converting date time from one presentation to another


[ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863265#comment-13863265
 ] 

Kostiantyn Kudriavtsev commented on HIVE-6046:
--

Hi guys! Could you please review these changes and proceed with it?

 add  UDF for converting date time from one presentation to another
 --

 Key: HIVE-6046
 URL: https://issues.apache.org/jira/browse/HIVE-6046
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Affects Versions: 0.13.0
Reporter: Kostiantyn Kudriavtsev
Assignee: Kostiantyn Kudriavtsev
 Attachments: hive-6046.patch


 it'd be nice to have function for converting datetime to different formats, 
 for example:
 format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
 There are two signatures to facilitate further using:
 format_date(datetime, fromFormat, toFormat)
 format_date(timestamp, toFormat)
  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HIVE-6006) Add UDF to calculate distance between geographic coordinates


 [ 
https://issues.apache.org/jira/browse/HIVE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kostiantyn Kudriavtsev resolved HIVE-6006.
--

  Resolution: Won't Fix
Release Note: According to discussion, proposed UDF is very specific to be 
included into common package

 Add UDF to calculate distance between geographic coordinates
 

 Key: HIVE-6006
 URL: https://issues.apache.org/jira/browse/HIVE-6006
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Affects Versions: 0.13.0
Reporter: Kostiantyn Kudriavtsev
Assignee: Kostiantyn Kudriavtsev
Priority: Minor
 Fix For: 0.13.0

 Attachments: hive-6006.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 It would be nice to have Hive UDF to calculate distance between two points on 
 Earth. Haversine formula seems to be good enough to overcome this issue
 The next function is proposed:
 HaversineDistance(lat1, lon1, lat2, lon2) - calculate Harvesine Distance 
 between 2 points with coordinates (lat1, lon1) and (lat2, lon2)



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Jason Dere

Congrats Vikram!

On Jan 6, 2014, at 11:10 AM, Prasanth Jayachandran 
pjayachand...@hortonworks.com wrote:

 Congratulations Vikram!!
 
 Thanks
 Prasanth Jayachandran
 
 On Jan 6, 2014, at 11:50 PM, Eugene Koifman ekoif...@hortonworks.com wrote:
 
 Congratulations!
 
 
 On Mon, Jan 6, 2014 at 9:44 AM, Gunther Hagleitner 
 ghagleit...@hortonworks.com wrote:
 
 Congratulations Vikram!
 
 Thanks,
 Gunther.
 
 
 On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan 
 hsubramani...@hortonworks.com wrote:
 
 congrats Vikram!!
 
 
 
 
 On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com
 wrote:
 
 Congrats Vikram!
 
 
 On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho jar...@apache.org
 wrote:
 Congratulations Vikram!
 
 Jarcec
 
 On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
 The Apache Hive PMC has voted to make Vikram Dixit a committer on the
 Apache Hive Project.
 
 Please join me in congratulating Vikram!
 
 Thanks.
 
 Carl
 
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
 entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
 reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.
 
 
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.
 
 
 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.
 
 
 -- 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to 
 which it is addressed and may contain information that is confidential, 
 privileged and exempt from disclosure under applicable law. If the reader 
 of this message is not the intended recipient, you are hereby notified that 
 any printing, copying, dissemination, distribution, disclosure or 
 forwarding of this communication is strictly prohibited. If you have 
 received this communication in error, please contact the sender immediately 
 and delete it from your system. Thank You.
 
 
 -- 
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to 
 which it is addressed and may contain information that is confidential, 
 privileged and exempt from disclosure under applicable law. If the reader 
 of this message is not the intended recipient, you are hereby notified that 
 any printing, copying, dissemination, distribution, disclosure or 
 forwarding of this communication is strictly prohibited. If you have 
 received this communication in error, please contact the sender immediately 
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-6134) Merging small files based on file size only works for CTAS queries

2014-01-06 Thread Eric Chu (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863292#comment-13863292
 ] 

Eric Chu commented on HIVE-6134:


Thanks [~ashutoshc] for pointing out the concatenate command. However, I think 
the ability to merge files for a table partition is orthogonal to supporting 
hive.merge.mapfiles, hive.merge.mapredfiles, and hive.merge.smallfiles.avgsize 
for regular queries (i.e., that don't result in a new table). Even if we have 
the optimal number of files at input for each partition, users querying over a 
large number of partitions with just SELECT FROM WHERE clauses will result in a 
large number of small output files, and there will be negative sides effects 
such as Hue timeout, the next job will have a large number of mappers, etc.

Can someone explain why the properties are supported only for queries with move 
tasks? Was it just a matter of scoping, or is there some reason that makes this 
inappropriate for queries without a move task? We are considering adding this 
support on our own and would like to get some insights on the original design 
considerations. Thanks!



 Merging small files based on file size only works for CTAS queries
 --

 Key: HIVE-6134
 URL: https://issues.apache.org/jira/browse/HIVE-6134
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.8.0, 0.10.0, 0.11.0, 0.12.0
Reporter: Eric Chu

 According to the documentation, if we set hive.merge.mapfiles to true, Hive 
 will launch an additional MR job to merge the small output files at the end 
 of a map-only job when the average output file size is smaller than 
 hive.merge.smallfiles.avgsize. Similarly, by setting hive.merge.mapredfiles 
 to true, Hive will merge the output files of a map-reduce job. 
 My expectation is that this is true for all MR queries. However, my 
 observation is that this is only true for CTAS queries. In 
 GenMRFileSink1.java, HIVEMERGEMAPFILES and HIVEMERGEMAPREDFILES are only used 
 if ((ctx.getMvTask() != null)  (!ctx.getMvTask().isEmpty())). So, for a 
 regular SELECT query that doesn't have move tasks, these properties are not 
 used.
 Is my understanding correct and if so, what's the reasoning behind the logic 
 of not supporting this for regular SELECT queries? It seems to me that this 
 should be supported for regular SELECT queries as well. One scenario where 
 this hits us hard is when users try to download the result in HUE, and HUE 
 times out b/c there are thousands of output files. The workaround is to 
 re-run the query as CTAS, but it's a significant time sink.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6134) Merging small files based on file size only works for CTAS queries

2014-01-06 Thread Xuefu Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863294#comment-13863294
 ] 

Xuefu Zhang commented on HIVE-6134:
---

[~ericchu30] Merging or concatenating files for a table/partition makes more 
sense in that the table/partition will likely be used over and over again. On 
the other hand, merging small files resulted from a query that are not 
permanently stored, while helping your case,  brings extra cost on the query 
execution, which is probably not a good idea for every query. If we choose to 
selectively, then the challenge is to know when to merge the result.

If the user has a better idea, then we can extend the Hive syntax to provide a 
construct such as SELECTM col1, col2 FROM table1,  but from your description, 
the users may not have that sense. They will not know until the query fails. 
Select and merge approach is close to your workaround of a temp table, right?

Having too many partitions pose many challenges including the problem you're 
facing. I'd suggest you revisit your partition strategy and try to reduce the 
number of partitions that a query would involve.

 Merging small files based on file size only works for CTAS queries
 --

 Key: HIVE-6134
 URL: https://issues.apache.org/jira/browse/HIVE-6134
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.8.0, 0.10.0, 0.11.0, 0.12.0
Reporter: Eric Chu

 According to the documentation, if we set hive.merge.mapfiles to true, Hive 
 will launch an additional MR job to merge the small output files at the end 
 of a map-only job when the average output file size is smaller than 
 hive.merge.smallfiles.avgsize. Similarly, by setting hive.merge.mapredfiles 
 to true, Hive will merge the output files of a map-reduce job. 
 My expectation is that this is true for all MR queries. However, my 
 observation is that this is only true for CTAS queries. In 
 GenMRFileSink1.java, HIVEMERGEMAPFILES and HIVEMERGEMAPREDFILES are only used 
 if ((ctx.getMvTask() != null)  (!ctx.getMvTask().isEmpty())). So, for a 
 regular SELECT query that doesn't have move tasks, these properties are not 
 used.
 Is my understanding correct and if so, what's the reasoning behind the logic 
 of not supporting this for regular SELECT queries? It seems to me that this 
 should be supported for regular SELECT queries as well. One scenario where 
 this hits us hard is when users try to download the result in HUE, and HUE 
 times out b/c there are thousands of output files. The workaround is to 
 re-run the query as CTAS, but it's a significant time sink.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5107) Change hive's build to maven

2014-01-06 Thread David Phillips (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Phillips updated HIVE-5107:
-

Description: I can not cope with hive's build infrastructure any more. I 
have started working on porting the project to maven. When I have some solid 
progess i will github the entire thing for review. Then we can talk about 
switching the project somehow.  (was:  I can not cope with hive's build 
infrastructure any more. I have started working on porting the project to 
maven. When I have some solid progess i will github the entire thing for 
review. Then we can talk about switching the project somehow.)

 Change hive's build to maven
 

 Key: HIVE-5107
 URL: https://issues.apache.org/jira/browse/HIVE-5107
 Project: Hive
  Issue Type: Task
Reporter: Edward Capriolo
Assignee: Edward Capriolo

 I can not cope with hive's build infrastructure any more. I have started 
 working on porting the project to maven. When I have some solid progess i 
 will github the entire thing for review. Then we can talk about switching the 
 project somehow.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5107) Change hive's build to maven

2014-01-06 Thread David Phillips (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Phillips updated HIVE-5107:
-

Description:  I can not cope with hive's build infrastructure any more. I 
have started working on porting the project to maven. When I have some solid 
progess i will github the entire thing for review. Then we can talk about 
switching the project somehow.  (was: I can not cope with hive's build 
infrastructure any more. I have started working on porting the project to 
maven. When I have some solid progess i will github the entire thing for 
review. Then we can talk about switching the project somehow.)

 Change hive's build to maven
 

 Key: HIVE-5107
 URL: https://issues.apache.org/jira/browse/HIVE-5107
 Project: Hive
  Issue Type: Task
Reporter: Edward Capriolo
Assignee: Edward Capriolo

  I can not cope with hive's build infrastructure any more. I have started 
 working on porting the project to maven. When I have some solid progess i 
 will github the entire thing for review. Then we can talk about switching the 
 project somehow.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6148) Support arbitrary structs stored in HBase

2014-01-06 Thread Swarnim Kulkarni (JIRA)

Swarnim Kulkarni created HIVE-6148:
--

 Summary: Support arbitrary structs stored in HBase
 Key: HIVE-6148
 URL: https://issues.apache.org/jira/browse/HIVE-6148
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 0.12.0
Reporter: Swarnim Kulkarni


We should add support to be able to query arbitrary structs stored in HBase.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6046) add UDF for converting date time from one presentation to another

2014-01-06 Thread Xuefu Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863307#comment-13863307
 ] 

Xuefu Zhang commented on HIVE-6046:
---

[~kostiantyn] Thanks for working on this. Could you please create a review 
board entry for your changes?

 add  UDF for converting date time from one presentation to another
 --

 Key: HIVE-6046
 URL: https://issues.apache.org/jira/browse/HIVE-6046
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Affects Versions: 0.13.0
Reporter: Kostiantyn Kudriavtsev
Assignee: Kostiantyn Kudriavtsev
 Attachments: hive-6046.patch


 it'd be nice to have function for converting datetime to different formats, 
 for example:
 format_date('2013-12-12 00:00:00.0', '-MM-dd HH:mm:ss.S', '/MM/dd')
 There are two signatures to facilitate further using:
 format_date(datetime, fromFormat, toFormat)
 format_date(timestamp, toFormat)
  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6067) Implement vectorized decimal comparison filters


 [ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6067:
--

Summary: Implement vectorized decimal comparison filters  (was: Implement 
vectorized decimal column-scalar comparison filters)

 Implement vectorized decimal comparison filters
 ---

 Key: HIVE-6067
 URL: https://issues.apache.org/jira/browse/HIVE-6067
 Project: Hive
  Issue Type: Sub-task
Reporter: Eric Hanson
Assignee: Eric Hanson
 Attachments: HIVE-6067.01.patch, HIVE-6067.02.patch


 Using the new DecimalColumnVector type, implement a template to generate 
 VectorExpression subclasses for Decimal comparison filters (, =, , =, =, 
 !=).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6067) Implement vectorized decimal comparison filters


[ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863310#comment-13863310
 ] 

Eric Hanson commented on HIVE-6067:
---

I'm generalizing this to cover column-scalar, scalar-column, and column-column 
comparison filters for decimal.

 Implement vectorized decimal comparison filters
 ---

 Key: HIVE-6067
 URL: https://issues.apache.org/jira/browse/HIVE-6067
 Project: Hive
  Issue Type: Sub-task
Reporter: Eric Hanson
Assignee: Eric Hanson
 Attachments: HIVE-6067.01.patch, HIVE-6067.02.patch


 Using the new DecimalColumnVector type, implement a template to generate 
 VectorExpression subclasses for Decimal comparison filters (, =, , =, =, 
 !=).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (HIVE-6149) TestJdbcDriver2 is unable to drop a database created from previous runs (hbasedb)


 [ 
https://issues.apache.org/jira/browse/HIVE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta reassigned HIVE-6149:
--

Assignee: Vaibhav Gumashta

 TestJdbcDriver2 is unable to drop a database created from previous runs 
 (hbasedb)
 ---

 Key: HIVE-6149
 URL: https://issues.apache.org/jira/browse/HIVE-6149
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor

 When tests are run sequentially on windows, the setUpBeforeClass method 
 attempts to drop all the previously created schemas and complains about being 
 unable to drop hbasedb. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6149) TestJdbcDriver2 is unable to drop a database created from previous runs (hbasedb)

Vaibhav Gumashta created HIVE-6149:
--

 Summary: TestJdbcDriver2 is unable to drop a database created from 
previous runs (hbasedb)
 Key: HIVE-6149
 URL: https://issues.apache.org/jira/browse/HIVE-6149
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Vaibhav Gumashta
Priority: Minor


When tests are run sequentially on windows, the setUpBeforeClass method 
attempts to drop all the previously created schemas and complains about being 
unable to drop hbasedb. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6149) TestJdbcDriver2 is unable to drop a database created from previous runs (hbasedb)


 [ 
https://issues.apache.org/jira/browse/HIVE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6149:
---

Affects Version/s: 0.13.0

 TestJdbcDriver2 is unable to drop a database created from previous runs 
 (hbasedb)
 ---

 Key: HIVE-6149
 URL: https://issues.apache.org/jira/browse/HIVE-6149
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Fix For: 0.13.0

 Attachments: HIVE-6149.1.patch


 When tests are run sequentially on windows, the setUpBeforeClass method 
 attempts to drop all the previously created schemas and complains about being 
 unable to drop hbasedb. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6149) TestJdbcDriver2 is unable to drop a database created from previous runs (hbasedb)


 [ 
https://issues.apache.org/jira/browse/HIVE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6149:
---

Fix Version/s: 0.13.0

 TestJdbcDriver2 is unable to drop a database created from previous runs 
 (hbasedb)
 ---

 Key: HIVE-6149
 URL: https://issues.apache.org/jira/browse/HIVE-6149
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Fix For: 0.13.0

 Attachments: HIVE-6149.1.patch


 When tests are run sequentially on windows, the setUpBeforeClass method 
 attempts to drop all the previously created schemas and complains about being 
 unable to drop hbasedb. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6149) TestJdbcDriver2 is unable to drop a database created from previous runs (hbasedb)


 [ 
https://issues.apache.org/jira/browse/HIVE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6149:
---

Attachment: HIVE-6149.1.patch

Actually there is no need to drop other databases since we are cleaning up the 
test tables and views before each test. 

 TestJdbcDriver2 is unable to drop a database created from previous runs 
 (hbasedb)
 ---

 Key: HIVE-6149
 URL: https://issues.apache.org/jira/browse/HIVE-6149
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Fix For: 0.13.0

 Attachments: HIVE-6149.1.patch


 When tests are run sequentially on windows, the setUpBeforeClass method 
 attempts to drop all the previously created schemas and complains about being 
 unable to drop hbasedb. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5441) Async query execution doesn't return resultset status


[ 
https://issues.apache.org/jira/browse/HIVE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863370#comment-13863370
 ] 

Vaibhav Gumashta commented on HIVE-5441:


[~thejas] [~prasadm] I think this slipped out of sight. Since it has already 
been +1-ed, I think we can get this in.

 Async query execution doesn't return resultset status
 -

 Key: HIVE-5441
 URL: https://issues.apache.org/jira/browse/HIVE-5441
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
 Fix For: 0.13.0

 Attachments: HIVE-5441.1.patch, HIVE-5441.3.patch, HIVE-5441.4.patch


 For synchronous statement execution (SQL as well as metadata and other), the 
 operation handle includes a boolean flag indicating whether the statement 
 returns a resultset. In case of async execution, that's always set to false.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Vaibhav Gumashta

Congrats Vikram!!!


On Mon, Jan 6, 2014 at 11:24 AM, Jason Dere jd...@hortonworks.com wrote:

 Congrats Vikram!

 On Jan 6, 2014, at 11:10 AM, Prasanth Jayachandran 
 pjayachand...@hortonworks.com wrote:

  Congratulations Vikram!!
 
  Thanks
  Prasanth Jayachandran
 
  On Jan 6, 2014, at 11:50 PM, Eugene Koifman ekoif...@hortonworks.com
 wrote:
 
  Congratulations!
 
 
  On Mon, Jan 6, 2014 at 9:44 AM, Gunther Hagleitner 
  ghagleit...@hortonworks.com wrote:
 
  Congratulations Vikram!
 
  Thanks,
  Gunther.
 
 
  On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan 
  hsubramani...@hortonworks.com wrote:
 
  congrats Vikram!!
 
 
 
 
  On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com
  wrote:
 
  Congrats Vikram!
 
 
  On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho 
 jar...@apache.org
  wrote:
  Congratulations Vikram!
 
  Jarcec
 
  On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
  The Apache Hive PMC has voted to make Vikram Dixit a committer on
 the
  Apache Hive Project.
 
  Please join me in congratulating Vikram!
 
  Thanks.
 
  Carl
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
  entity to
  which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
  reader
  of this message is not the intended recipient, you are hereby
 notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity
  to
  which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
 
 
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity
  to which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not

[jira] [Updated] (HIVE-5441) Async query execution doesn't return resultset status


 [ 
https://issues.apache.org/jira/browse/HIVE-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5441:



This has already been committed, marking as fixed.


 Async query execution doesn't return resultset status
 -

 Key: HIVE-5441
 URL: https://issues.apache.org/jira/browse/HIVE-5441
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 0.12.0
Reporter: Prasad Mujumdar
Assignee: Prasad Mujumdar
 Fix For: 0.13.0

 Attachments: HIVE-5441.1.patch, HIVE-5441.3.patch, HIVE-5441.4.patch


 For synchronous statement execution (SQL as well as metadata and other), the 
 operation handle includes a boolean flag indicating whether the statement 
 returns a resultset. In case of async execution, that's always set to false.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-3376) Local mode errors point to the wrong file


[ 
https://issues.apache.org/jira/browse/HIVE-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863376#comment-13863376
 ] 

Vaibhav Gumashta commented on HIVE-3376:


[~gangtimliu] Thanks for the work so far - this will be very useful while 
debugging. Are you still working on this JIRA? If not, I can give it a shot.

 Local mode errors point to the wrong file
 -

 Key: HIVE-3376
 URL: https://issues.apache.org/jira/browse/HIVE-3376
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability, Logging
Reporter: Gang Tim Liu
Assignee: Gang Tim Liu
 Attachments: hive-3371


 This is a classic example of a query that fails due to a bad udf in local 
 mode:
 Ended Job = job_local_0001 with errors
 Error during job, obtaining debugging information...
 Execution failed with exit status: 2
 Obtaining error information
 Task failed!
 Task ID:
 Stage-1
 Logs:
 /tmp/gang/hive.log
 FAILED: Execution Error, return code 2 from 
 org.apache.hadoop.hive.ql.exec.MapRedTask
 But actually local mr job in 
 /tmp/gang/gang_20120808181010_b716b5fb-5b8f-49d2-94e9-95bb8f465342.log



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Building Hive

2014-01-06 Thread Xuefu Zhang

You will need either -Phadoop-1 or -Phadoop-2 option.


On Mon, Jan 6, 2014 at 12:33 PM, Konstantin Weitz 
konstantin.we...@gmail.com wrote:

 I'm trying to build Hive from the trunk source. I ran

 mvn compile

 But I got the following error messages.
 What am I doing wrong?
 Is there documentation on how to build Hive?


 [INFO] -
 [INFO]
 
 [INFO] Reactor Summary:
 [INFO]
 [INFO] Hive .. SUCCESS [1.158s]
 [INFO] Hive Ant Utilities  SUCCESS [3.550s]
 [INFO] Hive Shims Common . SUCCESS [1.129s]
 [INFO] Hive Shims 0.20 ... SUCCESS [0.814s]
 [INFO] Hive Shims Secure Common .. SUCCESS [1.089s]
 [INFO] Hive Shims 0.20S .. SUCCESS [0.480s]
 [INFO] Hive Shims 0.23 ... SUCCESS [1.221s]
 [INFO] Hive Shims  SUCCESS [0.131s]
 [INFO] Hive Common ... FAILURE [2.518s]
 [INFO] Hive Serde  SKIPPED
 [INFO] Hive Metastore  SKIPPED
 [INFO] Hive Query Language ... SKIPPED
 [INFO] Hive Service .. SKIPPED
 [INFO] Hive JDBC . SKIPPED
 [INFO] Hive Beeline .. SKIPPED
 [INFO] Hive CLI .. SKIPPED
 [INFO] Hive Contrib .. SKIPPED
 [INFO] Hive HBase Handler  SKIPPED
 [INFO] Hive HCatalog . SKIPPED
 [INFO] Hive HCatalog Core  SKIPPED
 [INFO] Hive HCatalog Pig Adapter . SKIPPED
 [INFO] Hive HCatalog Server Extensions ... SKIPPED
 [INFO] Hive HCatalog Webhcat Java Client . SKIPPED
 [INFO] Hive HCatalog Webhcat . SKIPPED
 [INFO] Hive HCatalog HBase Storage Handler ... SKIPPED
 [INFO] Hive HWI .. SKIPPED
 [INFO] Hive ODBC . SKIPPED
 [INFO] Hive Shims Aggregator . SKIPPED
 [INFO] Hive TestUtils  SKIPPED
 [INFO] Hive Packaging  SKIPPED
 [INFO]
 
 [INFO] BUILD FAILURE
 [INFO]
 
 [INFO] Total time: 13.234s
 [INFO] Finished at: Mon Jan 06 12:23:33 PST 2014
 [INFO] Final Memory: 36M/186M
 [INFO]
 
 [ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
 (default-compile) on project hive-common: Compilation failure: Compilation
 failure:
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[44,30]
 package org.apache.hadoop.conf does not exist
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[46,32]
 package org.apache.hadoop.mapred does not exist
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[47,34]
 package org.apache.hadoop.security does not exist
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[48,30]
 package org.apache.hadoop.util does not exist
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[53,31]
 cannot find symbol
 [ERROR] symbol: class Configuration
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1063,31]
 cannot find symbol
 [ERROR] symbol:   class Configuration
 [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1068,32]
 cannot find symbol
 [ERROR] symbol:   class Configuration
 [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1081,33]
 cannot find symbol
 [ERROR] symbol:   class Configuration
 [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1086,33]
 cannot find symbol
 [ERROR] symbol:   class Configuration
 [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1090,33]
 cannot find symbol
 [ERROR] symbol:   class Configuration
 [ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
 [ERROR]
 hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1103,35]
 cannot find symbol
 [ERROR] symbol:   class

Building Hive

I'm trying to build Hive from the trunk source. I ran

mvn compile

But I got the following error messages.
What am I doing wrong?
Is there documentation on how to build Hive?


[INFO] -
[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] Hive .. SUCCESS [1.158s]
[INFO] Hive Ant Utilities  SUCCESS [3.550s]
[INFO] Hive Shims Common . SUCCESS [1.129s]
[INFO] Hive Shims 0.20 ... SUCCESS [0.814s]
[INFO] Hive Shims Secure Common .. SUCCESS [1.089s]
[INFO] Hive Shims 0.20S .. SUCCESS [0.480s]
[INFO] Hive Shims 0.23 ... SUCCESS [1.221s]
[INFO] Hive Shims  SUCCESS [0.131s]
[INFO] Hive Common ... FAILURE [2.518s]
[INFO] Hive Serde  SKIPPED
[INFO] Hive Metastore  SKIPPED
[INFO] Hive Query Language ... SKIPPED
[INFO] Hive Service .. SKIPPED
[INFO] Hive JDBC . SKIPPED
[INFO] Hive Beeline .. SKIPPED
[INFO] Hive CLI .. SKIPPED
[INFO] Hive Contrib .. SKIPPED
[INFO] Hive HBase Handler  SKIPPED
[INFO] Hive HCatalog . SKIPPED
[INFO] Hive HCatalog Core  SKIPPED
[INFO] Hive HCatalog Pig Adapter . SKIPPED
[INFO] Hive HCatalog Server Extensions ... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client . SKIPPED
[INFO] Hive HCatalog Webhcat . SKIPPED
[INFO] Hive HCatalog HBase Storage Handler ... SKIPPED
[INFO] Hive HWI .. SKIPPED
[INFO] Hive ODBC . SKIPPED
[INFO] Hive Shims Aggregator . SKIPPED
[INFO] Hive TestUtils  SKIPPED
[INFO] Hive Packaging  SKIPPED
[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time: 13.234s
[INFO] Finished at: Mon Jan 06 12:23:33 PST 2014
[INFO] Final Memory: 36M/186M
[INFO]

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
(default-compile) on project hive-common: Compilation failure: Compilation
failure:
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[44,30]
package org.apache.hadoop.conf does not exist
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[46,32]
package org.apache.hadoop.mapred does not exist
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[47,34]
package org.apache.hadoop.security does not exist
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[48,30]
package org.apache.hadoop.util does not exist
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[53,31]
cannot find symbol
[ERROR] symbol: class Configuration
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1063,31]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1068,32]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1081,33]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1086,33]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1090,33]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1103,35]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.conf.HiveConf
[ERROR]
hive/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1108,35]
cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler


[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863380#comment-13863380
 ] 

Brock Noland commented on HIVE-2599:


+1

 Support Composit/Compound Keys with HBaseStorageHandler
 ---

 Key: HIVE-2599
 URL: https://issues.apache.org/jira/browse/HIVE-2599
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, 
 HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt


 It would be really nice for hive to be able to understand composite keys from 
 an underlying HBase schema. Currently we have to store key fields twice to be 
 able to both key and make data available. I noticed John Sichi mentioned in 
 HIVE-1228 that this would be a separate issue but I cant find any follow up. 
 How feasible is this in the HBaseStorageHandler?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6050) JDBC backward compatibility is broken


[ 
https://issues.apache.org/jira/browse/HIVE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863384#comment-13863384
 ] 

Vaibhav Gumashta commented on HIVE-6050:


[~cwsteinbach] [~thejas] [~brocknoland] [~navis]
Do you guys have any thoughts on which would be a more common scenario to 
support: new jdbc driver compatible with old server version or old jdbc driver 
compatible with a newer server version?

 JDBC backward compatibility is broken
 -

 Key: HIVE-6050
 URL: https://issues.apache.org/jira/browse/HIVE-6050
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Szehon Ho
Priority: Blocker

 Connect from JDBC driver of Hive 0.13 (TProtocolVersion=v4) to HiveServer2 of 
 Hive 0.10 (TProtocolVersion=v1), will return the following exception:
 {noformat}
 java.sql.SQLException: Could not establish connection to 
 jdbc:hive2://localhost:1/default: Required field 'client_protocol' is 
 unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:336)
   at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:158)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at java.sql.DriverManager.getConnection(DriverManager.java:571)
   at java.sql.DriverManager.getConnection(DriverManager.java:187)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.getConnection(MyTestJdbcDriver2.java:73)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.lt;initgt;(MyTestJdbcDriver2.java:49)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
   at 
 org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:523)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1063)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:914)
 Caused by: org.apache.thrift.TApplicationException: Required field 
 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:327)
   ... 37 more
 {noformat}
 On code analysis, it looks like the 'client_protocol' scheme is a ThriftEnum, 
 which doesn't seem to be backward-compatible.  Look at the code path in the 
 generated file 'TOpenSessionReq.java', method 
 TOpenSessionReqStandardScheme.read():
 1. The method will call 'TProtocolVersion.findValue()' on the thrift 
 protocol's byte stream, which returns null if the client is sending an enum 
 value unknown to the server.  (v4 is unknown to server)
 2. The method will then call struct.validate(), which will throw the above 
 exception because of null version.  
 So doesn't look like the current backward-compatibility scheme will work.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Building Hive

Thanks, that got me a bit further. I ran

mvn compile -Phadoop-2

Unfortunately, now I get the following error. Any clue why?

[INFO]

[INFO] Building Hive Service 0.13.0-SNAPSHOT
[INFO]

[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] Hive .. SUCCESS [0.938s]
[INFO] Hive Ant Utilities  SUCCESS [1.288s]
[INFO] Hive Shims Common . SUCCESS [0.513s]
[INFO] Hive Shims 0.20 ... SUCCESS [0.410s]
[INFO] Hive Shims Secure Common .. SUCCESS [0.322s]
[INFO] Hive Shims 0.20S .. SUCCESS [0.178s]
[INFO] Hive Shims 0.23 ... SUCCESS [0.612s]
[INFO] Hive Shims  SUCCESS [0.097s]
[INFO] Hive Common ... SUCCESS [5.665s]
[INFO] Hive Serde  SUCCESS [9.090s]
[INFO] Hive Metastore  SUCCESS [17.912s]
[INFO] Hive Query Language ... SUCCESS [34.242s]
[INFO] Hive Service .. FAILURE [1.049s]
[INFO] Hive JDBC . SKIPPED
[INFO] Hive Beeline .. SKIPPED
[INFO] Hive CLI .. SKIPPED
[INFO] Hive Contrib .. SKIPPED
[INFO] Hive HBase Handler  SKIPPED
[INFO] Hive HCatalog . SKIPPED
[INFO] Hive HCatalog Core  SKIPPED
[INFO] Hive HCatalog Pig Adapter . SKIPPED
[INFO] Hive HCatalog Server Extensions ... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client . SKIPPED
[INFO] Hive HCatalog Webhcat . SKIPPED
[INFO] Hive HCatalog HBase Storage Handler ... SKIPPED
[INFO] Hive HWI .. SKIPPED
[INFO] Hive ODBC . SKIPPED
[INFO] Hive Shims Aggregator . SKIPPED
[INFO] Hive TestUtils  SKIPPED
[INFO] Hive Packaging  SKIPPED
[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time: 1:13.287s
[INFO] Finished at: Mon Jan 06 12:41:09 PST 2014
[INFO] Final Memory: 44M/458M
[INFO]

[ERROR] Failed to execute goal on project hive-service: Could not resolve
dependencies for project org.apache.hive:hive-service:jar:0.13.0-SNAPSHOT:
Could not find artifact org.apache.hive:hive-exec:jar:tests:0.13.0-SNAPSHOT
- [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR]   mvn goals -rf :hive-service



On Mon, Jan 6, 2014 at 12:36 PM, Xuefu Zhang xzh...@cloudera.com wrote:

 You will need either -Phadoop-1 or -Phadoop-2 option.


 On Mon, Jan 6, 2014 at 12:33 PM, Konstantin Weitz 
 konstantin.we...@gmail.com wrote:

  I'm trying to build Hive from the trunk source. I ran
 
  mvn compile
 
  But I got the following error messages.
  What am I doing wrong?
  Is there documentation on how to build Hive?
 
 
  [INFO] -
  [INFO]
  
  [INFO] Reactor Summary:
  [INFO]
  [INFO] Hive .. SUCCESS
 [1.158s]
  [INFO] Hive Ant Utilities  SUCCESS
 [3.550s]
  [INFO] Hive Shims Common . SUCCESS
 [1.129s]
  [INFO] Hive Shims 0.20 ... SUCCESS
 [0.814s]
  [INFO] Hive Shims Secure Common .. SUCCESS
 [1.089s]
  [INFO] Hive Shims 0.20S .. SUCCESS
 [0.480s]
  [INFO] Hive Shims 0.23 ... SUCCESS
 [1.221s]
  [INFO] Hive Shims  SUCCESS
 [0.131s]
  [INFO] Hive Common ... FAILURE
 [2.518s]

Re: Building Hive

2014-01-06 Thread Eugene Koifman

https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Howtobuildallsource
?


On Mon, Jan 6, 2014 at 12:42 PM, Konstantin Weitz 
konstantin.we...@gmail.com wrote:

 Thanks, that got me a bit further. I ran

 mvn compile -Phadoop-2

 Unfortunately, now I get the following error. Any clue why?

 [INFO]
 
 [INFO] Building Hive Service 0.13.0-SNAPSHOT
 [INFO]
 
 [INFO]
 
 [INFO] Reactor Summary:
 [INFO]
 [INFO] Hive .. SUCCESS [0.938s]
 [INFO] Hive Ant Utilities  SUCCESS [1.288s]
 [INFO] Hive Shims Common . SUCCESS [0.513s]
 [INFO] Hive Shims 0.20 ... SUCCESS [0.410s]
 [INFO] Hive Shims Secure Common .. SUCCESS [0.322s]
 [INFO] Hive Shims 0.20S .. SUCCESS [0.178s]
 [INFO] Hive Shims 0.23 ... SUCCESS [0.612s]
 [INFO] Hive Shims  SUCCESS [0.097s]
 [INFO] Hive Common ... SUCCESS [5.665s]
 [INFO] Hive Serde  SUCCESS [9.090s]
 [INFO] Hive Metastore  SUCCESS
 [17.912s]
 [INFO] Hive Query Language ... SUCCESS
 [34.242s]
 [INFO] Hive Service .. FAILURE [1.049s]
 [INFO] Hive JDBC . SKIPPED
 [INFO] Hive Beeline .. SKIPPED
 [INFO] Hive CLI .. SKIPPED
 [INFO] Hive Contrib .. SKIPPED
 [INFO] Hive HBase Handler  SKIPPED
 [INFO] Hive HCatalog . SKIPPED
 [INFO] Hive HCatalog Core  SKIPPED
 [INFO] Hive HCatalog Pig Adapter . SKIPPED
 [INFO] Hive HCatalog Server Extensions ... SKIPPED
 [INFO] Hive HCatalog Webhcat Java Client . SKIPPED
 [INFO] Hive HCatalog Webhcat . SKIPPED
 [INFO] Hive HCatalog HBase Storage Handler ... SKIPPED
 [INFO] Hive HWI .. SKIPPED
 [INFO] Hive ODBC . SKIPPED
 [INFO] Hive Shims Aggregator . SKIPPED
 [INFO] Hive TestUtils  SKIPPED
 [INFO] Hive Packaging  SKIPPED
 [INFO]
 
 [INFO] BUILD FAILURE
 [INFO]
 
 [INFO] Total time: 1:13.287s
 [INFO] Finished at: Mon Jan 06 12:41:09 PST 2014
 [INFO] Final Memory: 44M/458M
 [INFO]
 
 [ERROR] Failed to execute goal on project hive-service: Could not resolve
 dependencies for project org.apache.hive:hive-service:jar:0.13.0-SNAPSHOT:
 Could not find artifact org.apache.hive:hive-exec:jar:tests:0.13.0-SNAPSHOT
 - [Help 1]
 [ERROR]
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
 switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR]
 [ERROR] For more information about the errors and possible solutions,
 please read the following articles:
 [ERROR] [Help 1]

 http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
 [ERROR]
 [ERROR] After correcting the problems, you can resume the build with the
 command
 [ERROR]   mvn goals -rf :hive-service



 On Mon, Jan 6, 2014 at 12:36 PM, Xuefu Zhang xzh...@cloudera.com wrote:

  You will need either -Phadoop-1 or -Phadoop-2 option.
 
 
  On Mon, Jan 6, 2014 at 12:33 PM, Konstantin Weitz 
  konstantin.we...@gmail.com wrote:
 
   I'm trying to build Hive from the trunk source. I ran
  
   mvn compile
  
   But I got the following error messages.
   What am I doing wrong?
   Is there documentation on how to build Hive?
  
  
   [INFO] -
   [INFO]
  
 
   [INFO] Reactor Summary:
   [INFO]
   [INFO] Hive .. SUCCESS
  [1.158s]
   [INFO] Hive Ant Utilities  SUCCESS
  [3.550s]
   [INFO] Hive Shims Common . SUCCESS
  [1.129s]
   [INFO] Hive Shims 0.20 ... SUCCESS
  [0.814s]
   [INFO] Hive Shims Secure Common .. SUCCESS
  [1.089s]
   [INFO] Hive Shims 0.20S

[jira] [Updated] (HIVE-6150) Take advantage of Native HBase Compound keys


 [ 
https://issues.apache.org/jira/browse/HIVE-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-6150:
---

Description: Compound keys are now supported by HBase so we might be to 
integrate in a better way with HBase. See: 
https://issues.apache.org/jira/browse/HIVE-2599?focusedCommentId=13797506page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13797506

 Take advantage of Native HBase Compound keys
 

 Key: HIVE-6150
 URL: https://issues.apache.org/jira/browse/HIVE-6150
 Project: Hive
  Issue Type: Task
Reporter: Brock Noland

 Compound keys are now supported by HBase so we might be to integrate in a 
 better way with HBase. See: 
 https://issues.apache.org/jira/browse/HIVE-2599?focusedCommentId=13797506page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13797506



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5946) DDL authorization task factory should be better tested


[ 
https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863388#comment-13863388
 ] 

Thejas M Nair commented on HIVE-5946:
-

+1

 DDL authorization task factory should be better tested
 --

 Key: HIVE-5946
 URL: https://issues.apache.org/jira/browse/HIVE-5946
 Project: Hive
  Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-5946.patch, HIVE-5946.patch


 Thejas is working on various authorization issues and one element that might 
 be useful in that effort and increase test coverage and testability would be 
 perform authorization task creation in a factory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6150) Take advantage of Native HBase Compound keys

Brock Noland created HIVE-6150:
--

 Summary: Take advantage of Native HBase Compound keys
 Key: HIVE-6150
 URL: https://issues.apache.org/jira/browse/HIVE-6150
 Project: Hive
  Issue Type: Task
Reporter: Brock Noland






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler


[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863391#comment-13863391
 ] 

Brock Noland commented on HIVE-2599:


I created HIVE-6150 - Take advantage of Native HBase Compound keys for the 
changes to take advantage of HBASE-8693.

 Support Composit/Compound Keys with HBaseStorageHandler
 ---

 Key: HIVE-2599
 URL: https://issues.apache.org/jira/browse/HIVE-2599
 Project: Hive
  Issue Type: Improvement
  Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, 
 HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt


 It would be really nice for hive to be able to understand composite keys from 
 an underlying HBase schema. Currently we have to store key fields twice to be 
 able to both key and make data available. I noticed John Sichi mentioned in 
 HIVE-1228 that this would be a separate issue but I cant find any follow up. 
 How feasible is this in the HBaseStorageHandler?



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5946) DDL authorization task factory should be better tested


[ 
https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863392#comment-13863392
 ] 

Brock Noland commented on HIVE-5946:


Thanks Thejas! I will get an updated patch which fixes those tests in a few 
minutes.

 DDL authorization task factory should be better tested
 --

 Key: HIVE-5946
 URL: https://issues.apache.org/jira/browse/HIVE-5946
 Project: Hive
  Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-5946.patch, HIVE-5946.patch


 Thejas is working on various authorization issues and one element that might 
 be useful in that effort and increase test coverage and testability would be 
 perform authorization task creation in a factory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6100) Introduce basic set operations as UDFs

2014-01-06 Thread Alan Gates (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863394#comment-13863394
 ] 

Alan Gates commented on HIVE-6100:
--

INTERSECT, UNION, and EXCEPT are all SQL constructs.  Hive implements UNION ALL 
(only under very limited circumstances).  I'm saying if you are going to add 
these to Hive (and I agree they should be added) why not add them as part of 
the language itself, which is how most SQL users will expect them to be, rather 
than bolting them on with a UDF.

 Introduce basic set operations as UDFs
 --

 Key: HIVE-6100
 URL: https://issues.apache.org/jira/browse/HIVE-6100
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Kostiantyn Kudriavtsev
Priority: Minor
 Fix For: 0.13.0


 Introduce basic set operations:
 1. Intersection: The intersection of A and B, denoted by A ∩ B, is the set of 
 all things that are members of both A and B.
 select set_intersection(arr_a, arr_b) from dual
 2. Union: The union of A and B, denoted by A ∪ B, is the set of all things 
 that are members of either A or B.
 select set_union(arr_a, arr_b) from dual
 3. Symmetric difference: the symmetric difference of two sets is the set of 
 elements which are in either of the sets and not in their intersection.
 select set_symdiff(arr_a, arr_b) from dual



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5946) DDL authorization task factory should be better tested


 [ 
https://issues.apache.org/jira/browse/HIVE-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5946:
---

Attachment: HIVE-5946.patch

One line change was required in the TestHiveAuth* class.

 DDL authorization task factory should be better tested
 --

 Key: HIVE-5946
 URL: https://issues.apache.org/jira/browse/HIVE-5946
 Project: Hive
  Issue Type: Improvement
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-5946.patch, HIVE-5946.patch, HIVE-5946.patch


 Thejas is working on various authorization issues and one element that might 
 be useful in that effort and increase test coverage and testability would be 
 perform authorization task creation in a factory.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6100) Introduce basic set operations as UDFs

2014-01-06 Thread Edward Capriolo (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863402#comment-13863402
 ] 

Edward Capriolo commented on HIVE-6100:
---

I think Alan and I are speaking of two different things both of which are valid.

Form the title of the Jira I was assuming the user meant this.
{pre}
create table a ( listint x , list int y)
select union (x,y) 
{pre}

But what Alan is discussing is perfectly valid as well.

 Introduce basic set operations as UDFs
 --

 Key: HIVE-6100
 URL: https://issues.apache.org/jira/browse/HIVE-6100
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Reporter: Kostiantyn Kudriavtsev
Priority: Minor
 Fix For: 0.13.0


 Introduce basic set operations:
 1. Intersection: The intersection of A and B, denoted by A ∩ B, is the set of 
 all things that are members of both A and B.
 select set_intersection(arr_a, arr_b) from dual
 2. Union: The union of A and B, denoted by A ∪ B, is the set of all things 
 that are members of either A or B.
 select set_union(arr_a, arr_b) from dual
 3. Symmetric difference: the symmetric difference of two sets is the set of 
 elements which are in either of the sets and not in their intersection.
 select set_symdiff(arr_a, arr_b) from dual



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6050) JDBC backward compatibility is broken


[ 
https://issues.apache.org/jira/browse/HIVE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863408#comment-13863408
 ] 

Carl Steinbach commented on HIVE-6050:
--

I think running an older JDBC driver against a newer server version is going to 
be the more common scenario since there will always be cases of clients that 
are slow to upgrade.

 JDBC backward compatibility is broken
 -

 Key: HIVE-6050
 URL: https://issues.apache.org/jira/browse/HIVE-6050
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Szehon Ho
Priority: Blocker

 Connect from JDBC driver of Hive 0.13 (TProtocolVersion=v4) to HiveServer2 of 
 Hive 0.10 (TProtocolVersion=v1), will return the following exception:
 {noformat}
 java.sql.SQLException: Could not establish connection to 
 jdbc:hive2://localhost:1/default: Required field 'client_protocol' is 
 unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:336)
   at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:158)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at java.sql.DriverManager.getConnection(DriverManager.java:571)
   at java.sql.DriverManager.getConnection(DriverManager.java:187)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.getConnection(MyTestJdbcDriver2.java:73)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.lt;initgt;(MyTestJdbcDriver2.java:49)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
   at 
 org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:523)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1063)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:914)
 Caused by: org.apache.thrift.TApplicationException: Required field 
 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:327)
   ... 37 more
 {noformat}
 On code analysis, it looks like the 'client_protocol' scheme is a ThriftEnum, 
 which doesn't seem to be backward-compatible.  Look at the code path in the 
 generated file 'TOpenSessionReq.java', method 
 TOpenSessionReqStandardScheme.read():
 1. The method will call 'TProtocolVersion.findValue()' on the thrift 
 protocol's byte stream, which returns null if the client is sending an enum 
 value unknown to the server.  (v4 is unknown to server)
 2. The method will then call struct.validate(), which will throw the above 
 exception because of null version.  
 So doesn't look like the current backward-compatibility scheme will work.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6050) JDBC backward compatibility is broken


 [ 
https://issues.apache.org/jira/browse/HIVE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-6050:
-

Component/s: HiveServer2

 JDBC backward compatibility is broken
 -

 Key: HIVE-6050
 URL: https://issues.apache.org/jira/browse/HIVE-6050
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Szehon Ho
Priority: Blocker

 Connect from JDBC driver of Hive 0.13 (TProtocolVersion=v4) to HiveServer2 of 
 Hive 0.10 (TProtocolVersion=v1), will return the following exception:
 {noformat}
 java.sql.SQLException: Could not establish connection to 
 jdbc:hive2://localhost:1/default: Required field 'client_protocol' is 
 unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:336)
   at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:158)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at java.sql.DriverManager.getConnection(DriverManager.java:571)
   at java.sql.DriverManager.getConnection(DriverManager.java:187)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.getConnection(MyTestJdbcDriver2.java:73)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.lt;initgt;(MyTestJdbcDriver2.java:49)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
   at 
 org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:523)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1063)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:914)
 Caused by: org.apache.thrift.TApplicationException: Required field 
 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:327)
   ... 37 more
 {noformat}
 On code analysis, it looks like the 'client_protocol' scheme is a ThriftEnum, 
 which doesn't seem to be backward-compatible.  Look at the code path in the 
 generated file 'TOpenSessionReq.java', method 
 TOpenSessionReqStandardScheme.read():
 1. The method will call 'TProtocolVersion.findValue()' on the thrift 
 protocol's byte stream, which returns null if the client is sending an enum 
 value unknown to the server.  (v4 is unknown to server)
 2. The method will then call struct.validate(), which will throw the above 
 exception because of null version.  
 So doesn't look like the current backward-compatibility scheme will work.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Building Hive

Thank's Eugene, with that I would have known to use the -Phadoop=1 flag.
Using google I had only found the ant tutorial here
https://cwiki.apache.org/confluence/display/Hive/GettingStarted.

I think my current problem is that the build depends on version
0.13.0-SNAPSHOT of hive-exec that is not currently available online. The
latest version online is 0.12.0, see
http://mvnrepository.com/artifact/org.apache.hive/hive-exec

I saw that hive-exec is located in hive/ql, but I'm not able to
successfully build it. Do you know how to?



On Mon, Jan 6, 2014 at 12:44 PM, Eugene Koifman ekoif...@hortonworks.comwrote:


 https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Howtobuildallsource
 ?


 On Mon, Jan 6, 2014 at 12:42 PM, Konstantin Weitz 
 konstantin.we...@gmail.com wrote:

  Thanks, that got me a bit further. I ran
 
  mvn compile -Phadoop-2
 
  Unfortunately, now I get the following error. Any clue why?
 
  [INFO]
  
  [INFO] Building Hive Service 0.13.0-SNAPSHOT
  [INFO]
  
  [INFO]
  
  [INFO] Reactor Summary:
  [INFO]
  [INFO] Hive .. SUCCESS
 [0.938s]
  [INFO] Hive Ant Utilities  SUCCESS
 [1.288s]
  [INFO] Hive Shims Common . SUCCESS
 [0.513s]
  [INFO] Hive Shims 0.20 ... SUCCESS
 [0.410s]
  [INFO] Hive Shims Secure Common .. SUCCESS
 [0.322s]
  [INFO] Hive Shims 0.20S .. SUCCESS
 [0.178s]
  [INFO] Hive Shims 0.23 ... SUCCESS
 [0.612s]
  [INFO] Hive Shims  SUCCESS
 [0.097s]
  [INFO] Hive Common ... SUCCESS
 [5.665s]
  [INFO] Hive Serde  SUCCESS
 [9.090s]
  [INFO] Hive Metastore  SUCCESS
  [17.912s]
  [INFO] Hive Query Language ... SUCCESS
  [34.242s]
  [INFO] Hive Service .. FAILURE
 [1.049s]
  [INFO] Hive JDBC . SKIPPED
  [INFO] Hive Beeline .. SKIPPED
  [INFO] Hive CLI .. SKIPPED
  [INFO] Hive Contrib .. SKIPPED
  [INFO] Hive HBase Handler  SKIPPED
  [INFO] Hive HCatalog . SKIPPED
  [INFO] Hive HCatalog Core  SKIPPED
  [INFO] Hive HCatalog Pig Adapter . SKIPPED
  [INFO] Hive HCatalog Server Extensions ... SKIPPED
  [INFO] Hive HCatalog Webhcat Java Client . SKIPPED
  [INFO] Hive HCatalog Webhcat . SKIPPED
  [INFO] Hive HCatalog HBase Storage Handler ... SKIPPED
  [INFO] Hive HWI .. SKIPPED
  [INFO] Hive ODBC . SKIPPED
  [INFO] Hive Shims Aggregator . SKIPPED
  [INFO] Hive TestUtils  SKIPPED
  [INFO] Hive Packaging  SKIPPED
  [INFO]
  
  [INFO] BUILD FAILURE
  [INFO]
  
  [INFO] Total time: 1:13.287s
  [INFO] Finished at: Mon Jan 06 12:41:09 PST 2014
  [INFO] Final Memory: 44M/458M
  [INFO]
  
  [ERROR] Failed to execute goal on project hive-service: Could not resolve
  dependencies for project
 org.apache.hive:hive-service:jar:0.13.0-SNAPSHOT:
  Could not find artifact
 org.apache.hive:hive-exec:jar:tests:0.13.0-SNAPSHOT
  - [Help 1]
  [ERROR]
  [ERROR] To see the full stack trace of the errors, re-run Maven with the
 -e
  switch.
  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  [ERROR]
  [ERROR] For more information about the errors and possible solutions,
  please read the following articles:
  [ERROR] [Help 1]
 
 
 http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
  [ERROR]
  [ERROR] After correcting the problems, you can resume the build with the
  command
  [ERROR]   mvn goals -rf :hive-service
 
 
 
  On Mon, Jan 6, 2014 at 12:36 PM, Xuefu Zhang xzh...@cloudera.com
 wrote:
 
   You will need either -Phadoop-1 or -Phadoop-2 option.
  
  
   On Mon, Jan 6, 2014 at 12:33 PM, Konstantin Weitz 
   konstantin.we...@gmail.com wrote:
  
I'm trying to build Hive from the trunk source. I ran
   
mvn compile
   
But I got the

Re: [ANNOUNCE] New Hive Committer - Vikram Dixit

2014-01-06 Thread Vikram Dixit K

Thank you all for the support! I really appreciate it.

Thanks
Vikram.


On Mon, Jan 6, 2014 at 12:28 PM, Vaibhav Gumashta vgumas...@hortonworks.com
 wrote:

 Congrats Vikram!!!


 On Mon, Jan 6, 2014 at 11:24 AM, Jason Dere jd...@hortonworks.com wrote:

 Congrats Vikram!

 On Jan 6, 2014, at 11:10 AM, Prasanth Jayachandran 
 pjayachand...@hortonworks.com wrote:

  Congratulations Vikram!!
 
  Thanks
  Prasanth Jayachandran
 
  On Jan 6, 2014, at 11:50 PM, Eugene Koifman ekoif...@hortonworks.com
 wrote:
 
  Congratulations!
 
 
  On Mon, Jan 6, 2014 at 9:44 AM, Gunther Hagleitner 
  ghagleit...@hortonworks.com wrote:
 
  Congratulations Vikram!
 
  Thanks,
  Gunther.
 
 
  On Mon, Jan 6, 2014 at 9:33 AM, Hari Subramaniyan 
  hsubramani...@hortonworks.com wrote:
 
  congrats Vikram!!
 
 
 
 
  On Mon, Jan 6, 2014 at 9:22 AM, Thejas Nair the...@hortonworks.com
  wrote:
 
  Congrats Vikram!
 
 
  On Mon, Jan 6, 2014 at 9:01 AM, Jarek Jarcec Cecho 
 jar...@apache.org
  wrote:
  Congratulations Vikram!
 
  Jarcec
 
  On Mon, Jan 06, 2014 at 08:58:06AM -0800, Carl Steinbach wrote:
  The Apache Hive PMC has voted to make Vikram Dixit a committer on
 the
  Apache Hive Project.
 
  Please join me in congratulating Vikram!
 
  Thanks.
 
  Carl
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
  entity to
  which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
  reader
  of this message is not the intended recipient, you are hereby
 notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity
  to
  which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby
 notified
  that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
  immediately
  and delete it from your system. Thank You.
 
 
 
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity
  to which it is addressed and may contain information that is
 confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby
 notified that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.



 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to which it

[jira] [Commented] (HIVE-6050) JDBC backward compatibility is broken


[ 
https://issues.apache.org/jira/browse/HIVE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863426#comment-13863426
 ] 

Carl Steinbach commented on HIVE-6050:
--

It looks like Thrift IDL is not backward compatible wrt to Enums. We use Enums 
in other places in the IDL (e.g. TTypeId, TStatusCode, TOperationState, 
TOperationType, TGetTypeInfo, TFetchOrientation), and should probably 
investigate whether these references need to updated as well. I'm convinced 
that using an Enum for TGetTypeInfo was a bad idea, and suspect that the same 
may also be true for TTypeId.

 JDBC backward compatibility is broken
 -

 Key: HIVE-6050
 URL: https://issues.apache.org/jira/browse/HIVE-6050
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Szehon Ho
Priority: Blocker

 Connect from JDBC driver of Hive 0.13 (TProtocolVersion=v4) to HiveServer2 of 
 Hive 0.10 (TProtocolVersion=v1), will return the following exception:
 {noformat}
 java.sql.SQLException: Could not establish connection to 
 jdbc:hive2://localhost:1/default: Required field 'client_protocol' is 
 unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:336)
   at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:158)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at java.sql.DriverManager.getConnection(DriverManager.java:571)
   at java.sql.DriverManager.getConnection(DriverManager.java:187)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.getConnection(MyTestJdbcDriver2.java:73)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.lt;initgt;(MyTestJdbcDriver2.java:49)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
   at 
 org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:523)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1063)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:914)
 Caused by: org.apache.thrift.TApplicationException: Required field 
 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:327)
   ... 37 more
 {noformat}
 On code analysis, it looks like the 'client_protocol' scheme is a ThriftEnum, 
 which doesn't seem to be backward-compatible.  Look at the code path in the 
 generated file 'TOpenSessionReq.java', method 
 TOpenSessionReqStandardScheme.read():
 1. The method will call 'TProtocolVersion.findValue()' on the thrift 
 protocol's byte stream, which returns null if the client is sending an enum 
 value unknown to the server.  (v4 is unknown to server)
 2. The method will then call struct.validate(), which will throw the above 
 exception because of null version.  
 So doesn't look like the current backward-compatibility scheme will work.



--
This message was sent by

[jira] [Assigned] (HIVE-6050) JDBC backward compatibility is broken


 [ 
https://issues.apache.org/jira/browse/HIVE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach reassigned HIVE-6050:


Assignee: Carl Steinbach

 JDBC backward compatibility is broken
 -

 Key: HIVE-6050
 URL: https://issues.apache.org/jira/browse/HIVE-6050
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Szehon Ho
Assignee: Carl Steinbach
Priority: Blocker

 Connect from JDBC driver of Hive 0.13 (TProtocolVersion=v4) to HiveServer2 of 
 Hive 0.10 (TProtocolVersion=v1), will return the following exception:
 {noformat}
 java.sql.SQLException: Could not establish connection to 
 jdbc:hive2://localhost:1/default: Required field 'client_protocol' is 
 unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:336)
   at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:158)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at java.sql.DriverManager.getConnection(DriverManager.java:571)
   at java.sql.DriverManager.getConnection(DriverManager.java:187)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.getConnection(MyTestJdbcDriver2.java:73)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.lt;initgt;(MyTestJdbcDriver2.java:49)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
   at 
 org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:523)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1063)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:914)
 Caused by: org.apache.thrift.TApplicationException: Required field 
 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:327)
   ... 37 more
 {noformat}
 On code analysis, it looks like the 'client_protocol' scheme is a ThriftEnum, 
 which doesn't seem to be backward-compatible.  Look at the code path in the 
 generated file 'TOpenSessionReq.java', method 
 TOpenSessionReqStandardScheme.read():
 1. The method will call 'TProtocolVersion.findValue()' on the thrift 
 protocol's byte stream, which returns null if the client is sending an enum 
 value unknown to the server.  (v4 is unknown to server)
 2. The method will then call struct.validate(), which will throw the above 
 exception because of null version.  
 So doesn't look like the current backward-compatibility scheme will work.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Building Hive

Alright, I figured it out. You cannot use mvn compile, you have to use mvn
package and then it works! Thanks for all the support!


On Mon, Jan 6, 2014 at 1:00 PM, Konstantin Weitz konstantin.we...@gmail.com
 wrote:

 Thank's Eugene, with that I would have known to use the -Phadoop=1 flag.
 Using google I had only found the ant tutorial here
 https://cwiki.apache.org/confluence/display/Hive/GettingStarted.

 I think my current problem is that the build depends on version
 0.13.0-SNAPSHOT of hive-exec that is not currently available online. The
 latest version online is 0.12.0, see
 http://mvnrepository.com/artifact/org.apache.hive/hive-exec

 I saw that hive-exec is located in hive/ql, but I'm not able to
 successfully build it. Do you know how to?



 On Mon, Jan 6, 2014 at 12:44 PM, Eugene Koifman 
 ekoif...@hortonworks.comwrote:


 https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Howtobuildallsource
 ?


 On Mon, Jan 6, 2014 at 12:42 PM, Konstantin Weitz 
 konstantin.we...@gmail.com wrote:

  Thanks, that got me a bit further. I ran
 
  mvn compile -Phadoop-2
 
  Unfortunately, now I get the following error. Any clue why?
 
  [INFO]
  
  [INFO] Building Hive Service 0.13.0-SNAPSHOT
  [INFO]
  
  [INFO]
  
  [INFO] Reactor Summary:
  [INFO]
  [INFO] Hive .. SUCCESS
 [0.938s]
  [INFO] Hive Ant Utilities  SUCCESS
 [1.288s]
  [INFO] Hive Shims Common . SUCCESS
 [0.513s]
  [INFO] Hive Shims 0.20 ... SUCCESS
 [0.410s]
  [INFO] Hive Shims Secure Common .. SUCCESS
 [0.322s]
  [INFO] Hive Shims 0.20S .. SUCCESS
 [0.178s]
  [INFO] Hive Shims 0.23 ... SUCCESS
 [0.612s]
  [INFO] Hive Shims  SUCCESS
 [0.097s]
  [INFO] Hive Common ... SUCCESS
 [5.665s]
  [INFO] Hive Serde  SUCCESS
 [9.090s]
  [INFO] Hive Metastore  SUCCESS
  [17.912s]
  [INFO] Hive Query Language ... SUCCESS
  [34.242s]
  [INFO] Hive Service .. FAILURE
 [1.049s]
  [INFO] Hive JDBC . SKIPPED
  [INFO] Hive Beeline .. SKIPPED
  [INFO] Hive CLI .. SKIPPED
  [INFO] Hive Contrib .. SKIPPED
  [INFO] Hive HBase Handler  SKIPPED
  [INFO] Hive HCatalog . SKIPPED
  [INFO] Hive HCatalog Core  SKIPPED
  [INFO] Hive HCatalog Pig Adapter . SKIPPED
  [INFO] Hive HCatalog Server Extensions ... SKIPPED
  [INFO] Hive HCatalog Webhcat Java Client . SKIPPED
  [INFO] Hive HCatalog Webhcat . SKIPPED
  [INFO] Hive HCatalog HBase Storage Handler ... SKIPPED
  [INFO] Hive HWI .. SKIPPED
  [INFO] Hive ODBC . SKIPPED
  [INFO] Hive Shims Aggregator . SKIPPED
  [INFO] Hive TestUtils  SKIPPED
  [INFO] Hive Packaging  SKIPPED
  [INFO]
  
  [INFO] BUILD FAILURE
  [INFO]
  
  [INFO] Total time: 1:13.287s
  [INFO] Finished at: Mon Jan 06 12:41:09 PST 2014
  [INFO] Final Memory: 44M/458M
  [INFO]
  
  [ERROR] Failed to execute goal on project hive-service: Could not
 resolve
  dependencies for project
 org.apache.hive:hive-service:jar:0.13.0-SNAPSHOT:
  Could not find artifact
 org.apache.hive:hive-exec:jar:tests:0.13.0-SNAPSHOT
  - [Help 1]
  [ERROR]
  [ERROR] To see the full stack trace of the errors, re-run Maven with
 the -e
  switch.
  [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  [ERROR]
  [ERROR] For more information about the errors and possible solutions,
  please read the following articles:
  [ERROR] [Help 1]
 
 
 http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
  [ERROR]
  [ERROR] After correcting the problems, you can resume the build with the
  command
  [ERROR]   mvn goals -rf :hive-service
 
 
 
  On Mon, Jan 6, 2014 at 12:36 PM, Xuefu Zhang xzh...@cloudera.com
 wrote:
 
   You will need either

[jira] [Commented] (HIVE-6050) JDBC backward compatibility is broken


[ 
https://issues.apache.org/jira/browse/HIVE-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863439#comment-13863439
 ] 

Thejas M Nair commented on HIVE-6050:
-

bq. Do you guys have any thoughts on which would be a more common scenario to 
support: new jdbc driver compatible with old server version or old jdbc driver 
compatible with a newer server version?
I think it is very important that one version of jdbc driver be able to talk to 
different versions of HS2. If the older version of driver is able to talk to 
newer versions of HS2, that would be the ideal case.


 JDBC backward compatibility is broken
 -

 Key: HIVE-6050
 URL: https://issues.apache.org/jira/browse/HIVE-6050
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Reporter: Szehon Ho
Assignee: Carl Steinbach
Priority: Blocker

 Connect from JDBC driver of Hive 0.13 (TProtocolVersion=v4) to HiveServer2 of 
 Hive 0.10 (TProtocolVersion=v1), will return the following exception:
 {noformat}
 java.sql.SQLException: Could not establish connection to 
 jdbc:hive2://localhost:1/default: Required field 'client_protocol' is 
 unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:336)
   at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:158)
   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at java.sql.DriverManager.getConnection(DriverManager.java:571)
   at java.sql.DriverManager.getConnection(DriverManager.java:187)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.getConnection(MyTestJdbcDriver2.java:73)
   at 
 org.apache.hive.jdbc.MyTestJdbcDriver2.lt;initgt;(MyTestJdbcDriver2.java:49)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at 
 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
   at 
 org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
   at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
   at 
 org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
   at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:523)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1063)
   at 
 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:914)
 Caused by: org.apache.thrift.TApplicationException: Required field 
 'client_protocol' is unset! Struct:TOpenSessionReq(client_protocol:null)
   at 
 org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
   at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:160)
   at 
 org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:147)
   at 
 org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:327)
   ... 37 more
 {noformat}
 On code analysis, it looks like the 'client_protocol' scheme is a ThriftEnum, 
 which doesn't seem to be backward-compatible.  Look at the code path in the 
 generated file 'TOpenSessionReq.java', method 
 TOpenSessionReqStandardScheme.read():
 1. The method will call 'TProtocolVersion.findValue()' on the thrift 
 protocol's byte stream, which returns null if the client is sending an enum 
 value unknown to the server.  (v4 is unknown to server)
 2. The method will then call struct.validate(), which will throw the above 
 exception because of null version.  
 So doesn't look like the current backward-compatibility scheme will work.

[jira] [Updated] (HIVE-6125) Tez: Refactoring changes

2014-01-06 Thread Gunther Hagleitner (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6125:
-

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk.

 Tez: Refactoring changes
 

 Key: HIVE-6125
 URL: https://issues.apache.org/jira/browse/HIVE-6125
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Fix For: 0.13.0

 Attachments: HIVE-6125.1.patch, HIVE-6125.2.patch, HIVE-6125.3.patch, 
 HIVE-6125.4.patch


 In order to facilitate merge back I've separated out all the changes that 
 don't require Tez. These changes introduce new interfaces, move code etc. In 
 preparation of the Tez specific classes. This should help show what changes 
 have been made that affect the MR codepath as well.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler

2014-01-06 Thread Sushanth Sowmyan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863454#comment-13863454
 ] 

Sushanth Sowmyan commented on HIVE-6115:


Hi guys,

Looking through the code, there are two signatures for calls on 
addHbaseResources, one being 
addHBaseResources(Configuration,MapString,String) and the other being 
addHBaseResources(Configuration). The first function is implemented in the 
HiveHBaseStorageHandler itself, and the other is defined in HBaseConfiguration.

The one in HBaseConfiguration does the following:

   * adds resources (hbase config xmls) to the conf passed in to it
   * performs some checks.

The one in HiveHBaseStorageHandler does the following:

  * instantiates a new conf, and runs HBaseConfiguration.addHBaseResources on 
it.
  * Then, iterates through its properties, and for all values in it that are 
null (i.e. not present - covers only adds, not updates) in jobconf, it copies 
them into the MapString,String

So, our code path is relevant for the input side, where if called from 
hcatalog, we should be updating jobconf itself as well, so the extra bit makes 
sense. On the output side, however, where we do a copy into a copyOfConf, that 
looks redundant.



 Remove redundant code in HiveHBaseStorageHandler
 

 Key: HIVE-6115
 URL: https://issues.apache.org/jira/browse/HIVE-6115
 Project: Hive
  Issue Type: Improvement
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-6115.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler

2014-01-06 Thread Sushanth Sowmyan (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863459#comment-13863459
 ] 

Sushanth Sowmyan commented on HIVE-6115:


I'd love to have [~viraj] also comment - he might remember this refactor a bit 
better, I think the reason this redundancy was in is because we asked for a few 
other refactor changes that made the if-else structuring a bit more redundant 
than it used to be.

 Remove redundant code in HiveHBaseStorageHandler
 

 Key: HIVE-6115
 URL: https://issues.apache.org/jira/browse/HIVE-6115
 Project: Hive
  Issue Type: Improvement
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-6115.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Comment Edited] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler


[ 
https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863466#comment-13863466
 ] 

Brock Noland edited comment on HIVE-6115 at 1/6/14 9:47 PM:


Hi,

Thanks for this. The issue is that this code right here:

{noformat}
 if (this.configureInputJobProps) {
   try {
HBaseConfiguration.addHbaseResources(jobConf);
{noformat}

Adds hbase-default.xml as a resource to the jobConf and it does so after hbase 
configuration may already be added to the jobconf and thereby overrides any 
local configuration. Since HiveHBaseStorageHandler. addHBaseResources the 
relevant hbase configuration should already be present, my contention is that 
line should be removed.


was (Author: brocknoland):
Hi,

Thanks for this. The issue is that this code right here:

{noformat}
 if (this.configureInputJobProps) {
   try {
HBaseConfiguration.addHbaseResources(jobConf);
{noformat}

Adds hbase-default.xml as a resource to the jobConf and thereby overrides any 
local configuration. Since HiveHBaseStorageHandler. addHBaseResources the 
relevant hbase configuration should already be present, my contention is that 
line should be removed.

 Remove redundant code in HiveHBaseStorageHandler
 

 Key: HIVE-6115
 URL: https://issues.apache.org/jira/browse/HIVE-6115
 Project: Hive
  Issue Type: Improvement
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-6115.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6115) Remove redundant code in HiveHBaseStorageHandler


[ 
https://issues.apache.org/jira/browse/HIVE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863466#comment-13863466
 ] 

Brock Noland commented on HIVE-6115:


Hi,

Thanks for this. The issue is that this code right here:

{noformat}
 if (this.configureInputJobProps) {
   try {
HBaseConfiguration.addHbaseResources(jobConf);
{noformat}

Adds hbase-default.xml as a resource to the jobConf and thereby overrides any 
local configuration. Since HiveHBaseStorageHandler. addHBaseResources the 
relevant hbase configuration should already be present, my contention is that 
line should be removed.

 Remove redundant code in HiveHBaseStorageHandler
 

 Key: HIVE-6115
 URL: https://issues.apache.org/jira/browse/HIVE-6115
 Project: Hive
  Issue Type: Improvement
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
 Attachments: HIVE-6115.patch






--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6151) CLIService should use debug log level instead of info

Vaibhav Gumashta created HIVE-6151:
--

 Summary: CLIService should use debug log level instead of info
 Key: HIVE-6151
 URL: https://issues.apache.org/jira/browse/HIVE-6151
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta


The info level pollutes the production log.

cc [~hsubramaniyan] - thanks for spotting this.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Hive-trunk-hadoop2 - Build # 653 - Still Failing

Changes for Build #640

Changes for Build #641
[navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis 
reviewed by Thejas M Nair)

[navis] HIVE-4257 : java.sql.SQLNonTransientConnectionException on 
JDBCStatsAggregator (Teddy Choi via Navis, reviewed by Ashutosh)


Changes for Build #642

Changes for Build #643
[ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package 
from Microsoft to Hive (Hideaki Kumura via Eric Hanson)


Changes for Build #644
[cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming 
conventions (Sergey Shelukhin via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II 
(Navis via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis 
via cws)

[jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure 
vectorization produces same results as non-vectorized execution (Sergey 
Shelukhin via Jitendra Pandey)


Changes for Build #645

Changes for Build #646
[ehans] HIVE-5757: Implement vectorized support for CASE (Eric Hanson)


Changes for Build #647
[thejas] HIVE-5795 : Hive should be able to skip header and footer rows when 
reading data file for a table (Shuaishuai Nie via Thejas Nair)


Changes for Build #648
[thejas] HIVE-5923 : SQL std auth - parser changes (Thejas Nair, reviewed by 
Brock Noland)


Changes for Build #649

Changes for Build #650

Changes for Build #651
[brock] HIVE-3936 - Remote debug failed with hadoop 0.23X, hadoop 2.X (Swarnim 
Kulkarni via Brock)


Changes for Build #652

Changes for Build #653
[gunther] HIVE-6125: Tez: Refactoring changes (Gunther Hagleitner, reviewed by 
Thejas M Nair)




No tests ran.

The Apache Jenkins build system has built Hive-trunk-hadoop2 (build #653)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-hadoop2/653/ 
to view the results.

[jira] [Updated] (HIVE-6067) Implement vectorized decimal comparison filters


 [ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6067:
--

Description: Using the new DecimalColumnVector type, implement templates to 
generate VectorExpression subclasses for Decimal comparison filters (, =, , 
=, =, !=). Include scalar-column, column-scalar, and column-column filter 
cases. Include unit tests.  (was: Using the new DecimalColumnVector type, 
implement a template to generate VectorExpression subclasses for Decimal 
comparison filters (, =, , =, =, !=).)

 Implement vectorized decimal comparison filters
 ---

 Key: HIVE-6067
 URL: https://issues.apache.org/jira/browse/HIVE-6067
 Project: Hive
  Issue Type: Sub-task
Reporter: Eric Hanson
Assignee: Eric Hanson
 Attachments: HIVE-6067.01.patch, HIVE-6067.02.patch, 
 HIVE-6067.03.patch


 Using the new DecimalColumnVector type, implement templates to generate 
 VectorExpression subclasses for Decimal comparison filters (, =, , =, =, 
 !=). Include scalar-column, column-scalar, and column-column filter cases. 
 Include unit tests.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6151) CLIService should use debug log level instead of info


 [ 
https://issues.apache.org/jira/browse/HIVE-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6151:
---

Status: Patch Available  (was: Open)

 CLIService should use debug log level instead of info
 -

 Key: HIVE-6151
 URL: https://issues.apache.org/jira/browse/HIVE-6151
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Attachments: HIVE-6151.1.patch


 The info level pollutes the production log.
 cc [~hsubramaniyan] - thanks for spotting this.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6151) CLIService should use debug log level instead of info


 [ 
https://issues.apache.org/jira/browse/HIVE-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6151:
---

Priority: Minor  (was: Major)

 CLIService should use debug log level instead of info
 -

 Key: HIVE-6151
 URL: https://issues.apache.org/jira/browse/HIVE-6151
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Attachments: HIVE-6151.1.patch


 The info level pollutes the production log.
 cc [~hsubramaniyan] - thanks for spotting this.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6151) CLIService should use debug log level instead of info


 [ 
https://issues.apache.org/jira/browse/HIVE-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6151:
---

Attachment: HIVE-6151.1.patch

 CLIService should use debug log level instead of info
 -

 Key: HIVE-6151
 URL: https://issues.apache.org/jira/browse/HIVE-6151
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Attachments: HIVE-6151.1.patch


 The info level pollutes the production log.
 cc [~hsubramaniyan] - thanks for spotting this.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6067) Implement vectorized decimal comparison filters


 [ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6067:
--

Attachment: HIVE-6067.03.patch

Added cases for scalar-column and column-column comparison filters, with unit 
tests.

 Implement vectorized decimal comparison filters
 ---

 Key: HIVE-6067
 URL: https://issues.apache.org/jira/browse/HIVE-6067
 Project: Hive
  Issue Type: Sub-task
Reporter: Eric Hanson
Assignee: Eric Hanson
 Attachments: HIVE-6067.01.patch, HIVE-6067.02.patch, 
 HIVE-6067.03.patch


 Using the new DecimalColumnVector type, implement a template to generate 
 VectorExpression subclasses for Decimal comparison filters (, =, , =, =, 
 !=).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6149) TestJdbcDriver2 is unable to drop a database created from previous runs (hbasedb)


 [ 
https://issues.apache.org/jira/browse/HIVE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6149:
---

Status: Patch Available  (was: Open)

 TestJdbcDriver2 is unable to drop a database created from previous runs 
 (hbasedb)
 ---

 Key: HIVE-6149
 URL: https://issues.apache.org/jira/browse/HIVE-6149
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
Priority: Minor
 Fix For: 0.13.0

 Attachments: HIVE-6149.1.patch


 When tests are run sequentially on windows, the setUpBeforeClass method 
 attempts to drop all the previously created schemas and complains about being 
 unable to drop hbasedb. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Hive-trunk-h0.21 - Build # 2553 - Still Failing

Changes for Build #2539

Changes for Build #2540
[navis] HIVE-5414 : The result of show grant is not visible via JDBC (Navis 
reviewed by Thejas M Nair)


Changes for Build #2541

Changes for Build #2542
[ehans] HIVE-6017: Contribute Decimal128 high-performance decimal(p, s) package 
from Microsoft to Hive (Hideaki Kumura via Eric Hanson)


Changes for Build #2543
[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression II 
(Navis via cws)

[cws] HIVE-3746: Fix HS2 ResultSet Serialization Performance Regression (Navis 
via cws)

[jitendra] HIVE-6010: TestCompareCliDriver enables tests that would ensure 
vectorization produces same results as non-vectorized execution (Sergey 
Shelukhin via Jitendra Pandey)


Changes for Build #2544
[cws] HIVE-5911: Recent change to schema upgrade scripts breaks file naming 
conventions (Sergey Shelukhin via cws)


Changes for Build #2545

Changes for Build #2546
[ehans] HIVE-5757: Implement vectorized support for CASE (Eric Hanson)


Changes for Build #2547
[thejas] HIVE-5795 : Hive should be able to skip header and footer rows when 
reading data file for a table (Shuaishuai Nie via Thejas Nair)


Changes for Build #2548
[thejas] HIVE-5923 : SQL std auth - parser changes (Thejas Nair, reviewed by 
Brock Noland)


Changes for Build #2549

Changes for Build #2550

Changes for Build #2551
[brock] HIVE-3936 - Remote debug failed with hadoop 0.23X, hadoop 2.X (Swarnim 
Kulkarni via Brock)


Changes for Build #2552

Changes for Build #2553
[gunther] HIVE-6125: Tez: Refactoring changes (Gunther Hagleitner, reviewed by 
Thejas M Nair)




No tests ran.

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #2553)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/2553/ to 
view the results.

[jira] [Updated] (HIVE-6152) insert query fails on federation + viewfs

2014-01-06 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6152:
---

Summary: insert query fails on federation + viewfs  (was: insert query 
fails on federation + viewfs fails)

 insert query fails on federation + viewfs
 -

 Key: HIVE-6152
 URL: https://issues.apache.org/jira/browse/HIVE-6152
 Project: Hive
  Issue Type: Bug
Reporter: Ashutosh Chauhan

 This is because Hive first writes data to /tmp/ and than moves from /tmp to 
 final destination. In federated HDFS recommendation is to mount /tmp on a 
 separate nameservice, which is usually different than /user. Since renames 
 across different mount points are not supported, this fails. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-6152) insert query fails on federation + viewfs fails

2014-01-06 Thread Ashutosh Chauhan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HIVE-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6152:
---

Summary: insert query fails on federation + viewfs fails  (was: insert 
query fails on federation + viewfs)

 insert query fails on federation + viewfs fails
 ---

 Key: HIVE-6152
 URL: https://issues.apache.org/jira/browse/HIVE-6152
 Project: Hive
  Issue Type: Bug
Reporter: Ashutosh Chauhan

 This is because Hive first writes data to /tmp/ and than moves from /tmp to 
 final destination. In federated HDFS recommendation is to mount /tmp on a 
 separate nameservice, which is usually different than /user. Since renames 
 across different mount points are not supported, this fails. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HIVE-6152) insert query fails on federation + viewfs

2014-01-06 Thread Ashutosh Chauhan (JIRA)

Ashutosh Chauhan created HIVE-6152:
--

 Summary: insert query fails on federation + viewfs
 Key: HIVE-6152
 URL: https://issues.apache.org/jira/browse/HIVE-6152
 Project: Hive
  Issue Type: Bug
Reporter: Ashutosh Chauhan


This is because Hive first writes data to /tmp/ and than moves from /tmp to 
final destination. In federated HDFS recommendation is to mount /tmp on a 
separate nameservice, which is usually different than /user. Since renames 
across different mount points are not supported, this fails. 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5224) When creating table with AVRO serde, the avro.schema.url should be about to load serde schema from file system beside HDFS


[ 
https://issues.apache.org/jira/browse/HIVE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863491#comment-13863491
 ] 

Thejas M Nair commented on HIVE-5224:
-

[~shuainie] Can you please create a reviewboard link for the patch ?


 When creating table with AVRO serde, the avro.schema.url should be about to 
 load serde schema from file system beside HDFS
 

 Key: HIVE-5224
 URL: https://issues.apache.org/jira/browse/HIVE-5224
 Project: Hive
  Issue Type: Bug
Reporter: Shuaishuai Nie
Assignee: Shuaishuai Nie
 Attachments: HIVE-5224.1.patch, HIVE-5224.2.patch, HIVE-5224.4.patch, 
 Hive-5224.3.patch


 Now when loading schema for table with AVRO serde, the file system is hard 
 coded to hdfs in AvroSerdeUtils.java. This should enable loading schema from 
 file system beside hdfs.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6100) Introduce basic set operations as UDFs